KR101360999B1 - Real time data providing method and system based on augmented reality and portable terminal using the same - Google Patents

Abstract

The present invention relates to a method for providing real-time data based on augmented reality and a mobile terminal using the same, and more particularly, in real time by overlapping the data measured from an experimental sensor with an augmented reality in an image photographing an experimental situation such as a scientific experiment. It is to be provided.
In particular, the present invention by using the user's smart phone, tablet, such as a mobile phone by performing the photographing of the experimental situation, the generation of the augmented reality image overlapped with the measurement data and providing the measurement data, greatly improve the understanding and utility of the experimenter Can be improved.
In addition, the present invention enables the users of the relevant fields to collect and provide data according to various situations in real time even in the fields of remote medical diagnosis, broadcasting, architectural design, manufacturing process management, etc. There is an effect that can enable you to respond quickly.
Therefore, the reliability and competitiveness can be improved not only in the field of augmented reality, especially in the field of scientific experiments, but also in various fields such as remote medical diagnosis, broadcasting, architectural design, and manufacturing process management.

Description

Real time data providing method and system based on augmented reality based on augmented reality and portable terminal using the same}

The present invention relates to a method for providing real-time data based on augmented reality and a portable terminal using the same, and more particularly, in real time by overlapping the data measured from the sensor for augmented reality in an image photographing an experimental situation such as a scientific experiment. It is to be provided.

In particular, the present invention by using a mobile terminal of the user's smartphone, such as photographing the experimental situation, generation of augmented reality image overlapped with the measurement data and providing the measurement data, greatly improve the understanding and utility of the experimenter It relates to an augmented reality-based real-time data providing method and a portable terminal using the same.

In general, augmented reality is a technology that shows a virtual object (image) overlapping the real world that the user sees, and is used in various fields such as remote medical diagnosis, broadcasting, architectural design, and manufacturing process management.

The augmented reality field mainly includes location-based augmented reality using location sensors such as GPS / IGPS (Indoor GPS), image recognition-based augmented reality using QR codes or barcodes, and data-based augmented reality to display predetermined information as it is. It can be divided into

In other words, the existing augmented reality is provided in a manner that overlaps the predetermined static information in response to the set position information or recognition information is recognized or requested by the user. Here, the static information refers to information that cannot be actively changed once determined by the user (information changed only by the user), and can be used as opposed to dynamic information that is changed in real time, such as information input by a sensor. have.

Therefore, the field of augmented reality up to now is merely providing static information made by a user, and thus, the augmented reality field is not suitable for a field showing measurement data which is measured by a sensor and changed in real time in a laboratory of a scientific field. .

In the following prior art (Korea Patent Publication No. 10-1195446), a technique for providing a pre-stored drawing and maintenance guide information (static information) matched thereto based on augmented reality, but based on augmented reality based on dynamic information It does not appear about the technology provided by.

As a result, the augmented reality field that provides static information, like the prior art, can easily respond to a predetermined situation, but fails to actively respond to various situation changes (especially situations in which unexpected problems occur). Can cause.

On the other hand, the conventional scientific experiments stored the data measured by the sensor in the form of text, graphs, tables, etc., and documented the experimental process and data, and the analysis results of the data with supplementary explanations such as images as necessary. .

This process is cumbersome for documenting process such as setting up environment for actual experiments and recording, analyzing, and arranging results of experiments, and learning and using a separate program to prepare documents. In the case of complex experiments, in order to comprehend the process and improve the understanding of the non-experts, there are limitations in expression, although it requires a lot of cumbersome processes such as diagrams, videos, and images.

Republic of Korea Patent Publication No. 10-1195446 'Augmented reality based maintenance guide providing terminal and maintenance guide providing method using the same'

In order to solve the above problems, an object of the present invention is to provide a method for providing real-time data based on augmented reality that can actively provide data for various situations that are changed in real time, and a portable terminal using the same.

For example, in scientific experiments, a series of preparation procedures, sensor positions, experiment order, and expressions of changes in values detected by each sensor can be easily arranged and expressed to focus on experiments, enhance educational effects, and The purpose is to provide augmented reality-based real-time data providing method and a portable terminal using the same that can easily understand and share the experimental process and analysis results.

To this end, the present invention is a method for providing real-time data based on augmented reality that can be provided in real time by overlapping the data measured from the experimental sensor to the image photographing the experimental situation, such as scientific experiments, and a mobile terminal using the same The purpose is to provide.

In addition, the present invention by using a mobile terminal of the user's smart phone, tablet, etc. by performing the photographing of the experimental situation, the generation of the augmented reality image overlapped with the measurement data and providing the measurement data, greatly improved the understanding and utility of the experimenter An object of the present invention is to provide a method for providing augmented reality-based real-time data and a portable terminal using the same.

In order to achieve the above object, the augmented reality-based real-time data providing method according to the present invention, the target object setting step of setting at least one target object to be augmented reality-based data display; A target object checking step of checking a target object included in the original image among the target objects set by image recognition processing of the captured original image; A marker position checking step of tracking a position of a marker for displaying measurement data of the target object in the original image; A processed image generation step of generating a processed image by overlapping the measurement data for each object included in the original image with the original image so as to correspond to the marker position; And providing experimental information for providing at least one of the original image, the measurement data for each object, and the processed image as experimental data.

In addition, the marker position confirming step, the original image correction process for classifying the brightness component and the color component in the original image; A setting pixel extraction process of extracting a pixel having a color value set as the color of the marker; A pixel area checking process of classifying adjacent pixels among the extracted set pixels into one area; A marker region checking step of selecting a pixel region included in a set range among the classified pixel regions as a marker region; And a marker position tracking process for tracking the position of the selected marker region.

In addition, the portable terminal using the augmented reality-based real-time data providing method according to the present invention, the image taking unit for photographing the experimental situation in the original image; An image recognition unit to identify a target object set by image recognition processing of the original image photographed by the image photographing unit; A marker identification unit for tracking a marker corresponding to the target object in the original image processed by the image recognition unit; A virtual reality mapping unit for generating a processed image by overlapping the measurement data of the target object on the marker; And a display for outputting the processed image generated by the virtual reality mapping unit corresponding to a user's setting.

The apparatus may further include an object setting unit configured to transmit image information including at least one of an image, a color, and a size of at least one of the target object and the marker, which is the target of the experiment, to the image recognition unit in response to a user setting; And a measurement data receiver configured to transmit the measurement data measuring the change of the target object from the sensor installed in response to the experiment to the virtual reality mapping unit.

In addition, the data processing unit for receiving at least one of the original image, the measurement data for each object and the processed image from the virtual reality mapping unit, and converts at least one of the measurement data and the processed image for each target object corresponding to the user's setting ; A data storage unit for classifying the measured data for each target object, the processed image, and the original image converted by the data processing unit and storing them as a still image or a moving image; And a transmitting / receiving unit which transmits the measurement data for each target object, the processed image, and the original image converted or stored from the data processing unit or the data storage unit to an external device.

By the above solving means, the present invention can easily arrange and express a series of preparation process, the position of the sensor, the experiment order, the expression of the change of the value detected by each sensor, such as in the field of science experiment There is an advantage that can provide data actively.

As a result, the user can concentrate on the experiment, increase the educational effect, and can easily understand and share the experimental process and analysis results with the non-expert.

In particular, the present invention allows the user to check a variety of information as augmented reality-based image while photographing the experiment site in real time using a mobile terminal such as a smart phone, such as to provide the experimental results to the user as well as The generated augmented reality-based information has the advantage that can be used for education or various statistical data.

In addition, the present invention enables the users of the relevant fields to collect and provide data according to various situations in real time even in the fields of remote medical diagnosis, broadcasting, architectural design, manufacturing process management, etc. There is an effect that can enable you to respond quickly.

Therefore, the reliability and competitiveness can be improved not only in the field of augmented reality, especially in the field of scientific experiments, but also in various fields such as remote medical diagnosis, broadcasting, architectural design, and manufacturing process management.

1 is a flowchart illustrating an embodiment of a method for providing augmented reality-based real-time data according to the present invention.
FIG. 2 is a flowchart illustrating a specific embodiment of step S300 of FIG. 1.
3 is a block diagram illustrating an embodiment of an experimental data providing system using a mobile terminal using the augmented reality-based real-time data providing method according to the present invention.
4 is a block diagram illustrating an embodiment of the mobile terminal shown in FIG. 3.
5 is a view for explaining an embodiment of a processed image generated by the present invention.

Examples of a method for providing augmented reality-based real-time data and a mobile terminal using the same according to the present invention can be variously applied. Hereinafter, the most preferred embodiment will be described with reference to the accompanying drawings.

First, an embodiment of an augmented reality-based experimental data providing system according to the present invention will be described. Based on this, a method for providing augmented reality-based real-time data and a portable terminal will be described.

3 is a block diagram illustrating an embodiment of an experimental data providing system using a mobile terminal using the augmented reality-based real-time data providing method according to the present invention.

Referring to FIG. 3, the augmented reality-based experimental data providing system A may include a sensor 100, a mobile terminal 200, an experimental data providing server 300, and a client 400.

The sensor 100 is for measuring data required at various experimental sites (T), and may be variously applied according to the scientific experiment. For example, the sensor 100 may include a temperature sensor when measuring the temperature.

The mobile terminal 200 captures an image of an experimental site T (hereinafter, referred to as a photographed experimental field image), and receives measurement data transmitted from the sensor 100, and then augmented reality-based image (hereinafter, referred to as an image). , A processed image), and a configuration and a data processing method of the mobile terminal 200 will be described in detail below. Here, the mobile terminal 200 may include a smartphone, tablet, PDA, and the like.

The experimental data providing server 300 may receive and store the original image, the measurement data, and the processed image from the mobile terminal 200, and may provide the stored information in response to a request of the mobile terminal 200 and the client 400. .

In addition, the experimental data providing server 300 may receive and store measurement data from the sensor 100, and transmit the stored measurement data to the mobile terminal 200 at the request of the mobile terminal 200.

In other words, the mobile terminal 200 may receive measurement data directly from the sensor 100 or may be provided from the experimental data providing server 300. In addition, data can be directly obtained from a sensor embedded in the mobile terminal 200.

The client 400 may receive the original image, the measurement data, and the processed image stored in the experiment data providing server 300 to variously edit and utilize the image.

In addition, the client 400 can process the augmented reality-based real-time data providing method according to the present invention according to the needs of those skilled in the art, in this case, a separate camera is required to take the image of the experimental site (T) Of course.

Hereinafter, the augmented reality-based real-time data providing method applied to the experimental data providing system shown in FIG. 3 will be described in detail.

1 is a flowchart illustrating an embodiment of a method for providing augmented reality-based real-time data according to the present invention.

First, the mobile terminal 200 sets at least one target object to be an object for displaying augmented reality-based data (step S100). For example, as shown in FIG. 5, when measuring the temperature lowered by the liquid in the beaker and the room temperature, the target object may include a beaker Obj including the liquid.

The setting of the target object may be input by a user and made in the mobile terminal 200 through an image recognition algorithm. Here, it is obvious that the setting of the target object can be changed and applied in various ways according to the needs of those skilled in the art.

On the other hand, the method of displaying the measurement data in the original image includes a method of outputting the measurement data adjacent to the target object and a method of outputting the measurement data to a marker set by the user.

In the case of a method of outputting the measurement data adjacent to the target object, the tracking process of the marker to be described below may be omitted, and the output position may be directly designated by the user. When outputting the measurement data to the marker, it may be omitted to identify the target object to be described below.

In the present invention, when the target object is removed by the user (experimental) in the course of the experiment (when the target object is removed from the experiment site because it is not necessary for the subsequent experiment progress), the measurement data corresponding to the target object is marked In order to prevent it from appearing continuously, it will be described as recognizing a target object in the original image and outputting measurement data only to a marker corresponding to the target object included in the original image.

When the target object is set, the user can start the experiment, and when the experiment is started by the user, the mobile terminal 200 captures the experiment site and processes the image of the captured original image to be included in the original image of the set target object. Check the target object (step S200).

Then, the mobile terminal 200 tracks the position of a marker for displaying the measurement data of the target object in the original image (step S300). A method of tracking the position of the marker will be described in more detail below.

Subsequently, when the measurement data is received from the sensor S shown in FIG. 5, the mobile terminal 200 overlaps the measurement data for each object included in the original image to the original image to correspond to the marker position, thereby processing as shown in FIG. 5. An image is generated (step S400). The processed image is generated in various image forms such as text, graph, figure, etc. according to the sensor or experimental situation.

In FIG. 5, 'M1' is a marker corresponding to one target object Obj, 'M2' is a marker corresponding to another target object (indoor space), and 'D1' is any one target object. The temperature measured by Obj is measured and 'D2' is measured by measuring the temperature (indoor temperature) by another target object.

The mobile terminal 200 provides at least one of the original image, the measurement data for each object, and the processed image obtained by the above process to the at least one of the experimental data providing server 300 and the client 400 as experimental data. (Step S500).

FIG. 2 is a flowchart illustrating a specific embodiment of step S300 of FIG. 1.

First, the mobile terminal 200 may remove the brightness component from the original image (step S301). For example, the mobile terminal 200 may convert the original image into a YUV image and then remove a Y component (contrast) from the converted YUV image. Removal of the Y component can increase the recognition rate of the marker in a variety of indoor and outdoor experimental environments.

When the brightness component is removed from the original image, the mobile terminal 200 may extract a pixel having a color value set as the color of the marker (step S302). For example, the mobile terminal 200 may detect a corresponding color value from the coordinates of the UV image.

In addition, the mobile terminal 200 may classify adjacent pixels among the extracted set pixels into one independent area (step S303), and calculate an area for each area to be within a set range of the classified pixel areas. The included pixel area can be selected as the marker area (step S304). For example, 'M1' and 'M2' may be selected as markers by setting an area corresponding to a predetermined tolerance range with respect to the areas of 'M1' and 'M2' shown in FIG. 5.

Thereafter, the mobile terminal 200 can track the position of the marker region selected by the above process in real time with respect to the image photographed in real time (step S305), which is confirmed by tracking as shown in FIG. The measurement data can be overlapped at the position of the marker.

4 is a block diagram illustrating an embodiment of the mobile terminal shown in FIG. 3.

Referring to FIG. 4, the mobile terminal 200 for providing augmented reality-based real-time data according to the present invention includes an image photographing unit 210, an image recognition unit 220, a marker checker 230, and a virtual reality mapping unit. 240 and display 250.

The image capturing unit 210 captures the experimental situation of the scene as an original image, and may include a camera built in the mobile terminal 200. Of course, a separate camera can be installed by a person skilled in the art to photograph the experimental situation. In this case, the original image photographed by the camera can be provided to the mobile terminal 200.

The image recognizing unit 220 checks the target object set by image recognition processing of the original image photographed by the image capturing unit 210, requests the object setting unit 225 for image information on the target object, and receives the received image. The target object may be extracted from the original image based on the setting information.

The object setting unit 225 transmits image information including at least one of an image, a color, and a size of at least one of a target object and a marker that is a target of the experiment to the image recognition unit 220 in response to a user's setting. Can be. Here, the object setting unit 225 may transmit the stored image information at the request of the image recognition unit 220 while storing the image information. In addition, the object setting unit 225 may transmit the image information input by the user to the image recognition unit 220.

The marker checker 230 may track a marker corresponding to a target object included in the original image from the image recognized by the image recognizer 220, and may be transmitted from the image recognizer 220. Data about the original image, the processed target object, and the marker may be transmitted to the virtual reality mapping unit 240.

The virtual reality mapping unit 240 may generate the processed image as shown in FIG. 5 by overlapping the measurement data of the target object with the marker of the target object identified by the marker identification unit 230. In this case, the virtual reality mapping unit 240 may receive data measured by the sensor through the measurement data receiver 245.

The measurement data receiver 245 measures a change (eg, temperature change) of a target object from a sensor (eg, a temperature sensor) installed in response to the experiment, and measures the measured data in the virtual reality mapping unit 240. ) Can be sent.

The display 250 may output the processed image generated by the virtual reality mapping unit 240 according to a user's setting. In addition, the mobile terminal 200 for providing augmented reality-based real-time data according to the present invention may further include a data processor 260 and a data storage unit 270.

The data processor 260 converts the data transmitted from the virtual reality mapping unit 240 into various types of formats (still image, video, HTML), and stores the data in the data storage 270 or through the transceiver 265. The experimental data may be transmitted to an external device such as a server 300 or a client 400.

More specifically, the data processor 260 may receive at least one of the original image, the measurement data for each target object, and the processed image from the virtual reality mapping unit 240, and at least one of the received measurement data and the processed image for each target object. One can be converted corresponding to the user's setting.

The data storage unit 270 classifies and stores the measured data for each object converted from the data processing unit 260, the processed image, and the original image, and stores the original image as a video, and stores the original image as a video. It may include a measurement data storage module 272 for classifying the data into items for each target object and a processed image storage module 273 for storing the processed image on the basis of augmented reality.

In the above, the augmented reality based real-time data providing method and a portable terminal using the same have been described. It will be understood by those skilled in the art that the technical features of the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.

Therefore, it is to be understood that the embodiments described above are intended to be illustrative, and not restrictive, in all respects, and that the scope of the invention is indicated by the appended claims rather than the foregoing description, And all equivalents and modifications that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

100: Sensor
200: mobile terminal 210: video recording unit
220: image recognition unit 225: object setting unit
230: marker check unit 240: virtual reality mapping unit
245: measurement data receiver 250: display
260: data processor 265: transceiver
270: data storage unit 271: original image storage module
272: measurement data storage module 273: processed image storage module
300: experimental data providing server
400: client

Claims (5)

A target object setting step of setting at least one target object as a target for displaying augmented reality-based data;
A target object checking step of checking a target object included in the original image among the target objects set by image recognition processing of the captured original image;
A marker position checking step of tracking a position of a marker for displaying measurement data of the target object in the original image;
A processed image generation step of generating a processed image by overlapping the measurement data for each object included in the original image with the original image so as to correspond to the marker position; And
Experimental information providing step of providing at least one of the original image, the measurement data for each object and the processed image as experimental data,
The marker position checking step,
An original image correction process of removing brightness components from the original image;
A setting pixel extraction process of extracting a pixel having a color value set as the color of the marker;
A pixel area checking process of classifying adjacent pixels among the extracted set pixels into one area;
A marker region checking step of selecting a pixel region included in a set range among the classified pixel regions as a marker region; And
Augmented reality-based real-time data providing method comprising a marker position tracking process for tracking the position of the selected marker region.
delete Image capturing unit for photographing the experimental situation in the original image;
An image recognition unit to identify a target object set by image recognition processing of the original image photographed by the image photographing unit;
A marker identification unit for tracking a marker corresponding to the target object in the original image processed by the image recognition unit;
A virtual reality mapping unit for generating a processed image by overlapping the measurement data of the target object on the marker; And
It includes a display for outputting the processed image generated by the virtual reality mapping unit corresponding to the user's setting,
A data processor configured to receive at least one of the original image, the measurement data for each object, and the processed image from the virtual reality mapping unit, and convert at least one of the measurement data for each object and the processed image according to a user's setting;
A data storage unit for classifying the measured data for each target object, the processed image, and the original image converted by the data processing unit and storing them as a still image or a moving image; And
Using the augmented reality-based real-time data providing method further comprises a transmission and reception unit for transmitting the measurement data, processed image and the original image for each target object converted or stored from the data processing unit or the data storage unit to an external device. Mobile terminal.
The method of claim 3, wherein
An object setting unit configured to transmit image information including at least one of an image, a color, and a size of at least one of the target object and the marker to be the target of the experiment in response to a user setting; And
And a measurement data receiver for transmitting the measurement data measuring the change of the target object from the sensor installed in response to the experiment to the virtual reality mapping unit.
delete

Images