JP5682815B2 - Information processing apparatus, image forming apparatus, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, an image forming apparatus, and a program for transmitting information related to an image forming apparatus to a portable terminal device for display.

  In recent years, printers and copiers used for business printing have been greatly increased in speed and accordingly increased in size. For example, in the printing industry, high-speed machines having a printing speed of 100 sheets or more per minute and large-scale machines having a length from one end of the apparatus to the other end of as much as 10 meters are becoming popular.

  If the image forming apparatus is increased in speed and size as described above, the number of recording sheets conveyed simultaneously in the apparatus during the printing process becomes very large, and if a jam (paper jam) occurs, recovery work is difficult. become.

  For example, when a jam occurs on the downstream side of the paper transport path, such as near the paper discharge port, in a model in which the number of simultaneously transporting recording papers during the printing process is close to 20, the jammed paper present in the apparatus The number of recording sheets) may be as many as 10 or more. An operator involved in the printing work takes out a large number of these jammed papers one by one while confirming the jam position displayed on the operation panel, and it is difficult to remove the jammed paper. In particular, when a large machine with a length of 10 meters is used, when an operator removes jammed paper scattered in the device in order while looking at the operation panel, there is jammed paper that cannot be seen from the outside. Or the operation panel is at a position away from the operator, which is a very complicated and labor-intensive operation.

  As a technique for reducing such a jam paper removal operation, for example, in Patent Document 1, an operation unit of a print processing apparatus is separated from an apparatus main body, and a transmission signal from a transmitter provided in the operation unit is used. Technology that receives information from multiple receivers provided at necessary locations on the device body, determines the position of the operation unit, and displays information (such as information for recovering from jams) on the operation unit according to the position of the operation unit Is disclosed.

JP 2006-231675 A

  In the configuration in which the position of the operation unit (general terminal device) separated from the apparatus main body is detected by the position detection means including the transmitter and the receiver as in Patent Document 1, the operation unit is disposed on the apparatus main body side. It is necessary to provide a large number of position detection means (a plurality of receivers in the example of Patent Document 1) for detecting the position (three-dimensional position), and there is a problem that costs increase.

  The present invention is intended to solve the above-described problem. When information related to an image forming apparatus is transmitted to a portable terminal device and displayed, the information is displayed according to the position of the portable terminal device. An object of the present invention is to provide an information processing apparatus, an image forming apparatus, and a program that can be realized with a simple configuration.

  The gist of the present invention for achieving the object lies in the inventions of the following items.

[1] A communication unit that communicates with a mobile terminal having a photographing function, a display function, and a communication function;
The portable terminal receives a photographed image obtained by photographing the image forming apparatus from the portable terminal , determines a photographing part photographed by the portable terminal of the image forming apparatus based on the photographed image, and determines the photographing part. A control unit for transmitting corresponding display information to the portable terminal;
An information processing apparatus comprising:

In the above invention, based on a photographed image of the image forming apparatus by the portable terminal, a photographing part photographed by the portable terminal of the image forming apparatus is determined, and display information corresponding to the photographing part is transmitted to the portable terminal. Display. As described above, if the display information corresponding to the shooting location is displayed on the mobile terminal using the captured image, a large number of position detecting means for detecting the position of the mobile terminal is unnecessary and simple. Appropriate display information (display information corresponding to the shooting location) can be displayed on the portable terminal with a simple configuration.

[2] The control unit determines a viewpoint position of the mobile terminal that captured the captured image based on the captured image, and transmits display information of an orientation corresponding to the viewpoint position to the mobile terminal. The information processing apparatus according to [1].

  In the above invention, since the display information in the direction corresponding to the viewpoint position is displayed on the mobile terminal, the display information can be displayed in an easy-to-understand manner for the operator who operates the mobile terminal from the viewpoint position.

[3] The information processing apparatus according to [1] or [2], wherein the control unit synthesizes the display information with the captured image and transmits the synthesized image to the mobile terminal.

  In the above invention, since the display information is synthesized and displayed on the photographed image, the operator who operates the mobile terminal can see the relationship between the actual photographing location (actual) in the photographed image and the display information. It can be displayed easily.

[4] The control unit transmits, to the portable terminal, display information corresponding to an event occurring in the image forming apparatus and to be notified to the portable terminal, and the shooting location. The information processing apparatus according to any one of to [3].

  In the above invention, when an event to be notified to the mobile terminal occurs in the image forming apparatus, the display information corresponding to the event and the shooting location of the image forming apparatus by the mobile terminal (event occurring in the image forming apparatus) Display information corresponding to the shooting location) can be displayed on the portable terminal.

[5] having an input unit for inputting designated place information indicating a designated place where an event to be notified to the mobile terminal occurs in the image forming apparatus;
When the designated location indicated by the designated location information input from the input unit is included in the shooting location, the control unit transmits first display information corresponding to the designated location to the portable terminal. And when it is not contained in the said imaging | photography location, the 2nd display information corresponding to the designated location is transmitted to the said portable terminal. The information processing apparatus as described in [4] characterized by the above-mentioned.

  In the above invention, the display information displayed on the portable terminal is made different depending on whether or not the portable terminal is photographing the designated portion of the image forming apparatus. For example, if you are shooting a specified location, display direct display information about the specified location, and if you are not shooting the specified location, display indirect display information about the specified location, Different display information can be used. In this way, various display information can be displayed on the mobile terminal and provided to the operator of the mobile terminal.

[6] The information processing apparatus according to [5], wherein the second display information is guidance information to the designated location.

  In the above invention, when the shooting location by the mobile terminal is out of the designated location, the guidance information to the designated location is displayed, so that the operator of the mobile terminal can easily reach the designated location.

[7] The display information is at least one of position information indicating a position of a predetermined portion of the image forming apparatus, a fluoroscopic image of the image forming apparatus, and operation information of the image forming apparatus. The information processing apparatus according to any one of [1] to [6], which is characterized.

  In the above invention, the mobile terminal includes position information indicating the position of the predetermined position of the image forming apparatus, a fluoroscopic image of the image forming apparatus, and operation information of the image forming apparatus as display information corresponding to the shooting position of the image forming apparatus. And at least one of them is displayed. The operator of the portable terminal can easily grasp the position of the predetermined portion of the image forming apparatus from the position information. Further, the internal state of the image forming apparatus can be grasped from the fluoroscopic image. Further, the operation content of the image forming apparatus can be easily grasped from the operation information.

[8] The display information is information related to at least one of an error occurrence location, a maintenance location, and an operation required location in the image forming apparatus. [1] to [7] The information processing apparatus according to any one of the above.

  In the above invention, information (display information) related to at least one of the error occurrence location, the parts replacement location, and the operation required location in the image forming apparatus is displayed on the portable terminal. The operator of the portable terminal can easily cope with and deal with the occurrence of errors, replacement of parts, and necessary operations in the image forming apparatus from the information displayed on the portable terminal.

[9] On the outer surface of the image forming apparatus, a predetermined mark for determining the viewpoint position of the mobile terminal that images the image forming apparatus is provided.
The information processing apparatus according to [2], wherein the control unit determines a viewpoint position of the mobile terminal based on deformation of the predetermined mark included in the captured image.

  In the above invention, the viewpoint position of the mobile terminal can be accurately determined by using a predetermined mark provided on the outer surface of the image forming apparatus (based on deformation of the predetermined mark).

[10] The control unit receives, in place of the photographed image, photographing location information indicating a photographing location photographed by the portable terminal in the image forming apparatus obtained by the portable terminal based on the photographed image. The information processing apparatus according to [1], wherein display information corresponding to a shooting location of the image forming apparatus indicated by the shooting location information is transmitted to the portable terminal.

  In the said invention, a portable terminal calculates | requires the imaging | photography location of an image forming apparatus based on a picked-up image. By obtaining the photographing location information indicating the photographing location from the portable terminal, display information corresponding to the photographing location of the image forming apparatus indicated by the photographing location information can be transmitted to the portable terminal.

[11] The control unit receives viewpoint position information indicating the viewpoint position of the mobile terminal that has captured the captured image, which is obtained by the mobile terminal based on the captured image, and the viewpoint indicated by the viewpoint position information. The information processing apparatus according to [10], wherein display information having an orientation corresponding to a position is transmitted to the mobile terminal.

In the said invention, a portable terminal calculates | requires the viewpoint position of the portable terminal which image | photographed the picked-up image based on a picked-up image. By obtaining the viewpoint position information indicating the viewpoint position from the portable terminal, it is possible to transmit the display information of the direction corresponding to the viewpoint position indicated by the viewpoint position information to the portable terminal.
[12] The control unit determines whether or not a location where an event to be notified to the mobile terminal occurring in the image forming apparatus occurs is included in the shooting location, and the location where the event occurs Is included in the shooting location, the first display information is transmitted to the mobile terminal as the display information, and when the location where the event has occurred is not included in the shooting location, the display information The information processing apparatus according to any one of [1] to [3], wherein second display information different from the first display information is transmitted to the portable terminal.
[13] The control unit determines whether or not a location where an event to be notified to the mobile terminal occurring in the image forming apparatus occurs is included in the shooting location, and the location where the event occurs Any one of [1] to [3], wherein a skeleton image corresponding to the shooting location is transmitted to the portable terminal as the display information when the shooting location is included in the shooting location. Information processing device.
[14] The control unit determines whether or not a location where an event that should be notified to the mobile terminal occurring in the image forming apparatus occurs is included in the shooting location, and a location where the event occurs [1] to [3], [13], wherein guidance information to the location where the event has occurred is transmitted to the portable terminal as the display information. The information processing apparatus according to any one of the above.
[15] The control unit includes, as the display information, a skeleton image corresponding to the shooting location and guidance information to a location where an event to be notified to the mobile terminal occurring in the image forming apparatus occurs . The information processing apparatus according to any one of [1] to [3], wherein the information processing apparatus transmits the information to the portable terminal.

[ 16 ] The information processing apparatus according to any one of [1] to [ 15 ];
A detection unit that detects the occurrence of an event to be notified to the mobile terminal in its own device,
The said control part transmits the display information according to the event which the said detection part detected, and the said imaging | photography location to the said portable terminal. The image forming apparatus characterized by the above-mentioned.

  In the above invention, the occurrence of an event to be notified to the mobile terminal by the image forming apparatus is detected, and display information corresponding to the detected event and a shooting location by the mobile terminal is transmitted to the mobile terminal and displayed. Can be provided to operators.

[17] In the information processing apparatus,
A function of communicating with a mobile terminal having a shooting function, a display function, and a communication function;
Receiving said imaged image obtained by imaging a mobile terminal Gae image forming apparatus from the portable terminal determines the imaging position taken by the mobile terminal of the image forming apparatus based on the captured image, the imaging position A function of transmitting display information according to the mobile terminal,
A program to realize
[18] In the function of transmitting display information corresponding to the shooting location to the portable terminal, does the shooting location include a location where an event to be notified to the portable terminal occurring in the image forming apparatus has occurred? When the location where the event has occurred is included in the shooting location, the first display information is transmitted to the mobile terminal as the display information, and the location where the event has occurred is [17] The program according to [17], wherein when it is not included in a shooting location, second display information different from the first display information is transmitted to the portable terminal as the display information.
[19] In the function of transmitting display information corresponding to the shooting location to the portable terminal, is the shooting location included a location where an event to be notified to the portable terminal occurring in the image forming apparatus has occurred? If the location where the event occurs is included in the shooting location, a skeleton image corresponding to the shooting location is transmitted to the mobile terminal as the display information. 17].
[20] In the function of transmitting display information corresponding to the shooting location to the portable terminal, whether the shooting location includes a location where an event to be notified to the portable terminal occurring in the image forming apparatus has occurred When the location where the event has occurred is not included in the shooting location, guidance information to the location where the event has occurred is transmitted to the portable terminal as the display information. The program according to [17] or [19].
[21] In the function of transmitting display information corresponding to the shooting location to the portable terminal, the display information is notified to the skeleton image corresponding to the shooting location and the portable terminal generated in the image forming apparatus. The program according to [17], wherein guide information to a place where a power event has occurred is transmitted to the portable terminal.

[ 22 ] In a portable terminal having a photographing function, a display function, and a communication function,
A function for obtaining a photographing location photographed by the mobile terminal of the image forming device based on a photographed image obtained by photographing the image forming device, and transmitting photographing location information indicating the photographing location to the information processing device;
A function of receiving, from the information processing device, display information corresponding to a shooting location of the image forming apparatus indicated by the shooting location information;
A function of displaying display information received from the information processing apparatus;
A program to realize

[ 23 ] A function of obtaining a viewpoint position of the mobile terminal that has captured the captured image based on the captured image, and transmitting viewpoint position information indicating the viewpoint position to the information processing apparatus;
A function of receiving display information of an orientation corresponding to the viewpoint position of the mobile terminal indicated by the viewpoint position information from the information processing apparatus;
The program according to [22] , wherein the portable terminal is implemented.

  According to the information processing apparatus, the image forming apparatus, and the program according to the present invention, when information related to the image forming apparatus is transmitted to the portable terminal device and displayed, the information corresponding to the position of the portable terminal device is displayed. The display can be realized with a simple configuration.

1 is a block diagram illustrating a schematic configuration of a multifunction peripheral (image forming apparatus) according to a first embodiment of the present invention. FIG. 3 is a diagram illustrating an example of shooting of a multifunction peripheral by a mobile terminal and a display example of the shot image. It is a figure which shows the state which synthesize | combined and displayed the information which shows the position of a jam position on the picked-up image of the multifunctional device by a portable terminal. It is a figure which shows the state which synthesize | combined and displayed the skeleton image which shows the position and condition of jam virtually on the picked-up image of the multifunction device by a portable terminal. It is a figure which shows the state which synthesize | combined and displayed the information which shows the direction of a jam position on the picked-up image of the multifunctional device by a portable terminal. FIG. 6 is a diagram illustrating an example of a method for specifying a viewpoint position of a mobile terminal that captures an image of a multifunction device using four marks provided on the multifunction device. FIG. 7 is a diagram schematically showing how the four marks in FIG. 6 appear when the multifunction machine is viewed from diagonally above the front side. It is the figure which showed typically the state of the two marks which are reflected in each picked-up image when the two marks in FIG. 6 are imaged at different distances and angles. It is a figure explaining an example of the method of calculating | requiring the angle of a portable terminal based on the arrangement | positioning space | interval and distance of four marks which appeared in the picked-up image. It is a sequence diagram which shows the flow of a process of the multifunctional device and portable terminal which concern on the 1st Embodiment of this invention. It is a sequence diagram which shows the flow of a process of the multifunctional device and portable terminal which concern on the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 shows a schematic configuration of a multifunction machine 10 as an image forming apparatus according to the first embodiment of the present invention. The multi-function device 10 has a copy function for optically reading an image of a document, printing the duplicate image on a recording paper and outputting it, storing the image data of the read document as a file, an external terminal such as a personal computer, or a server A printer function that prints an image based on print data received from an external terminal or an image based on image data stored in the multifunction machine 10 on a recording sheet, and a facsimile function. ing.

  The multifunction machine 10 includes a central processing unit (CPU) 11 that performs overall control of the operation of the multifunction machine 10, a read only memory (ROM) 12 connected to the CPU 11, a random access memory (RAM) 13, and a nonvolatile memory 14. A hard disk device 15, a display unit 16, an operation unit 17, a facsimile communication unit 18, a network communication unit 19, a scanner unit 20, an image processing unit 21, a printer unit 22, and a post-processing unit 23. It has.

  The CPU 11 is based on an OS program, on which middleware, application programs, and the like are executed. Various programs are stored in the ROM 12, and the functions of the multifunction machine 10 such as job execution are realized by the CPU 11 executing processes according to these programs. The RAM 13 is used as a work memory for temporarily storing various data when the CPU 11 executes a program, an image memory for storing image data, and the like. Other necessary programs are loaded from the hard disk device 15 to the RAM 13 and executed.

  The nonvolatile memory 14 is a rewritable memory (flash memory) that retains memory even when the power is turned off. The nonvolatile memory 14 stores device-specific information (device ID (IDentification), IP (Internet Protocol) address, etc.), various setting information, and the like. The hard disk device 15 is a large-capacity nonvolatile storage device, and is used for storing print data and image data in addition to programs.

  The operation panel of the multifunction machine 10 includes a display unit 16 and an operation unit 17. The display unit 16 is configured by a liquid crystal display or the like, and displays various screens such as a main screen, a menu screen, an operation screen, a setting screen, and an information display screen. The operation unit 17 receives various operations from the user. The operation unit 17 includes a touch panel that is provided on the screen of the display unit 16 and detects a pressed coordinate position, and includes a numeric keypad, a character input key, a start key, and the like.

  The facsimile communication unit 18 controls operations related to facsimile transmission and reception. The network communication unit 19 controls operations related to communication through a network such as a LAN (Local Area Network).

  The scanner unit 20 optically reads an image of a document and acquires image data. The scanner unit 20 sequentially moves, for example, a light source that irradiates light on a document, a line image sensor that receives the reflected light for one line in the width direction, and a reading position in line units in the length direction of the document. An optical system composed of a lens, a mirror, and the like for guiding the reflected light from the document to the line image sensor to form an image, a conversion unit for converting the analog image signal output from the line image sensor into digital image data, etc. It is configured with.

  The image processing unit 21 performs various kinds of image processing such as image correction, rotation, enlargement / reduction, compression / expansion on the image data.

  The printer unit 22 forms and outputs an image based on the image data on a recording sheet by an electrophotographic process. The printer unit 22 includes, for example, a recording paper transport device, a photosensitive drum, a charging device, an LD (Laser Diode) that is controlled to be turned on according to input image data, and laser light emitted from the LD. It is configured as a so-called laser printer having a scanning unit that scans on a photosensitive drum, a developing device, a transfer separating device, a cleaning device, and a fixing device. Instead of an LED printer that irradiates a photosensitive drum with an LED (Light Emitting Diode) instead of a laser beam, other types of printers may be used.

  The post-processing unit 23 performs post-processing (finishing processing) such as stapling, punching, folding, and bookbinding on the recording paper printed out from the printer unit 22.

  The multifunction machine 10 has a function of detecting the occurrence of various events determined in advance, a function of detecting the occurrence location of the event, and automatically operating the event and the detection location or operating the operation unit 17. The function of receiving and displaying on the display unit 16 is provided. Various events include errors such as jams (paper jams) and failures, the necessity of maintenance such as inspection and replacement of consumable parts, and the necessity of manual operation of the panel and lever associated with the errors and maintenance. In detecting these event occurrence locations, locations to be detected, that is, error occurrence detection locations, maintenance necessity detection locations (maintenance locations), manual operation necessity detection locations associated with errors and maintenance (manual operation locations) Is also predetermined. Further, the occurrence of errors and the necessity of maintenance (maintenance timing) are detected by a sensor or a program.

  In addition, the multifunction machine 10 has a function of communicating with a mobile terminal (general terminal device) with a camera in which a dedicated application program for communicating with the own apparatus is installed, and a function of the own apparatus captured by the mobile terminal. It has a function of receiving a photographed image from a portable terminal and transmitting display information corresponding to the photographing location of the own device specified based on the photographed image to the portable terminal for display.

  The display information is information related to a location where the occurrence of the above-described event (error, maintenance necessity, manual operation accompanying the error or maintenance) is detected. Of a plurality of predetermined detection target locations, a location where a detection target event has occurred and actually detected is a designated location (predetermined location) where display information should be transmitted to the mobile terminal. The display information is information relating to the designated location. Specifically, information (position information) indicating the position of the designated location, image information (skeleton image) showing the designated location as a perspective (frame), information for guiding to the designated location (guidance information), designated location The information (operation information) indicating the operation content of.

  The image data of the display information is stored in a database stored in the hard disk device 15 (or the nonvolatile memory 14). The position information, guidance information, and operation information are created and stored in advance using, for example, characters, symbols, figures, marks, and the like. The skeleton image uses 3D CAD data of a skeleton image (perspective image / framework image) of a necessary location (a plurality of detection target locations) created by 3D CAD (Computer Aided Design) at the time of designing the MFP 10. The three-dimensional CAD data is stored.

  Further, various display information (position information / skeleton image / operation information) excluding the guidance information is stored in the database in association with the detection target portion. For example, a three-dimensional coordinate system is set in the internal space of the multifunction machine 10, and various displays corresponding to the detection target location are displayed at the three-dimensional coordinate position (X, Y, Z) of the detection target location in the three-dimensional coordinate system. Information is associated and stored in the database. The three-dimensional coordinate position (center position) of the detection target portion may be the center of gravity position of the detection target portion, or may be an arbitrarily set reference position.

  FIG. 2 shows an example of photographing of the multifunction machine 10 by the portable terminal 30 and display of the photographed image. Specifically, a state is shown in which the user operates the portable terminal 30 to photograph the multifunction machine 10 with the camera function of the portable terminal 30 and the captured image is displayed on the display unit 31 of the portable terminal 30.

  In this embodiment, the portable terminal 30 is configured by a general-purpose device such as a smartphone in which a cellular phone and a personal digital assistant (PDA) are integrated. The mobile terminal 30 may be a general-purpose mobile phone or mobile information terminal having a shooting function (camera function), a display function, and a communication function, or a dedicated mobile phone having such functions. The mobile terminal may be attached to the multifunction device 10 and the dedicated mobile terminal may be used. In addition, the dedicated portable terminal may have either wireless communication or wired communication.

  A dedicated application program for communicating with the multifunction machine 10 is installed in the general-purpose portable terminal 30. The portable terminal 30 activates this application program to communicate with the multifunction device 10, transmits a photographed image of the multifunction device 10 photographed by the camera unit to the multifunction device 10, and displays display information received from the multifunction device 10 as a display unit. 31.

  Next, shooting of the multifunction device 10 and display of information by the mobile terminal 30 will be described. Here, an example will be described in which a jam occurs in the multifunction machine 10 and display information regarding the jam occurrence location is displayed on the mobile terminal 30. Also, in the following, (1) display examples of various display information, (2) an example of a method for specifying a shooting location, (3) an example of a method for determining display information according to the shooting location, and (4) the position of the mobile terminal and the shooting location An example of how to create display information according to the relationship will be described in order.

(1) Display Example of Various Display Information As shown in FIG. 2, the user holds the portable terminal 30 with his hand, and the camera function (camera unit) of the portable terminal 30 surrounds the multifunction device 10 (in the example of FIG. 2, the multifunction device). An arbitrary part of the multifunction machine 10 is photographed from the front side of the machine 10. The portable terminal 30 displays on the display unit 31 a photographed image of an arbitrary part of the multifunction machine 10 photographed from the viewpoint position of the camera unit.

  FIG. 3 shows a state in which a jam position image 50 that is display information indicating a location where a jam has occurred is superimposed and displayed on a captured image of a specific location of the multifunction machine 10 by the mobile terminal 30. In this example, the jam position image 50 is composed of numbers (numbers).

  FIG. 4 shows a skeleton image 60, a jam position image 50, and a virtual image of the internal state of the jam occurrence location and the jam paper as seen through the captured image of a specific location of the multifunction machine 10 by the mobile terminal 30. Shows a state in which are superimposed and displayed.

  In FIG. 5, a jam direction image 70, which is display information indicating the direction of the jam occurrence location (the direction in which the jam paper exists), is superimposed on the captured image of the location other than the jam occurrence location of the MFP 10 by the mobile terminal 30. Is displayed. In this example, the jam direction image 70 is composed of an arrow mark indicating the direction of the jam occurrence location and characters (“This Direction”).

(2) Example of Method for Specifying Shooting Location The shooting location of the multifunction machine 10 by the mobile terminal 30 is specified based on the captured image. A known image collation / determination technique is used for specifying (determining) a shooting location based on the captured image.

  For example, a database of collation information is stored in the hard disk device 15 (or the nonvolatile memory 14) of the multifunction device 10, and an image (image data) of a characteristic portion of the multifunction device 10 to be collated is stored in the database. Are stored in association with each other. The feature part may use various shape parts or outlines that can recognize the boundary with the surroundings based on outlines or colors, or may use predetermined marks provided on the outer surface of the multifunction device 10. Also good.

  In the image matching, an image of a feature portion is extracted from a photographed image of the multifunction machine 10 by the mobile terminal 30, and the image of the feature portion is compared with an image of a feature portion for matching in a database to identify a matching one. . From the three-dimensional coordinate position associated with the image of the feature part matched in the database, and the position and size of the feature part (extracted feature part) in the photographed image, the composite image shown in the photographed image The shooting location of the machine 10 is specified by the range of three-dimensional coordinates.

(3) Example of determination method of display information according to shooting location As described above, a detection target location of an event to be detected (error, maintenance necessity, manual operation accompanying error or maintenance) in the multifunction machine 10 Various display information (position information / skeleton image / operation information) excluding guide information is associated with (three-dimensional coordinate position) and stored in the database. The location where the detection target event has occurred becomes a designated location (predetermined location) where display information should be transmitted to the mobile terminal 30, and the display information corresponding to this designated location is selected and transmitted to the mobile terminal 30.

  Here, when the specified photographing location (three-dimensional coordinate range) includes a designated portion (three-dimensional coordinate position), position information (jam position image 50), skeleton image (skeleton image 60), Operation information or the like is determined as display information to be transmitted to the mobile terminal 30. When the specified shooting location does not include the specified location, the guide information (such as an arrow mark indicating the direction from the shooting location to the specified location) according to the shooting location is transmitted to the mobile terminal 30. Display information to be determined.

  In addition, when there are a plurality of designated locations, only the guidance information corresponding to the latest designated location may be determined as a transmission target, or each guidance information corresponding to all the designated locations is targeted for transmission. It may be determined.

(4) Example of display information creation method according to the positional relationship between the mobile terminal and the shooting location When the display information is a skeleton image, the direction according to the viewpoint position of the camera unit of the mobile terminal 30 (the direction seen from the viewpoint position) Generate a skeleton image of This viewpoint position is also specified based on the captured image of the multifunction machine 10 by the portable terminal 30. Specifically, a feature part included in the captured image is extracted, and a viewpoint position is specified based on the feature part. As the characteristic portion, the above-described various shape portions, contours, and the like may be used, or a predetermined mark provided on the outer surface of the multifunction machine 10 may be used.

  6 to 9 illustrate a method for specifying the viewpoint position using the mark. As shown in FIGS. 6 and 7, a plurality of black marks 80 serving as characteristic portions are provided on the outer surface (front surface in the illustrated example) of the multifunction machine 10. Information such as the size (including shape) and arrangement interval (D1) of these marks 80 is registered in advance in the nonvolatile memory 14 or the like.

  When a portion including the mark 80 is photographed by the portable terminal 30 from the front side of the multifunction machine 10, an image of the mark 80 appears in the photographed image. For example, an image of the mark 80 as shown in FIGS. 8 and 9 appears. The image of the mark 80 is extracted as a feature portion, and the camera portion of the mobile terminal 30 is detected from the shooting location based on the size, shape (distortion angle), arrangement interval, and pre-registered information of the extracted mark 80 image. The distance and angle (up / down / left / right angle) are calculated, and the viewpoint position of the camera unit is obtained. That is, based on how the image size, shape (distortion angle), arrangement interval, and the like of the extracted mark 80 are deformed with respect to the actual mark 80 provided in the multifunction device 10, The distance and angle (up / down / left / right angle) from the mobile terminal 30 to the camera unit are calculated, and the viewpoint position of the camera unit is obtained.

  For example, when the image of the four marks 80 illustrated in FIG. 9 appears in the photographed image, the ratio between the horizontal arrangement interval (D1) and the arrangement interval (D1 ′) on the photographed image according to the registration information, Based on the ratio of the vertical distance d1 between the two images and the vertical distance d2 between the two right images, the angles of the actual four marks 80 provided on the multifunction device 10 and the camera unit of the mobile terminal 30 are determined. Can be sought.

  As in the above example, the viewpoint position of the camera unit of the mobile terminal 30 is specified, and a skeleton image that is viewed from the viewpoint position is generated from the three-dimensional CAD data stored in the database.

  Next, operations of the multifunction machine 10 and the portable terminal 30 will be described.

  FIG. 10 is a sequence diagram illustrating a processing flow of the multifunction machine 10 and the portable terminal 30.

  In response to an operation from the user, the portable terminal 30 activates a dedicated application program for communicating with the multifunction machine 10 (P101). When the application program is activated, the IP address of the multifunction device 10 is designated to identify the communication partner multifunction device 10.

  The portable terminal 30 (dedicated application program) inquires of the multifunction device 10 whether or not the communication partner multifunction device 10 can currently accept communication (P102). At this time, authentication information registered in the portable terminal 30 may be transmitted at the same time, and stored as user authentication data on the multifunction device 10 side.

  Upon receiving an inquiry from the portable terminal 30, the multifunction device 10 determines whether or not the own device is currently communicable (P103). For example, if a fatal error has occurred in the device itself and communication cannot be accepted, it is determined that the device cannot be accepted. Here, a case will be described where it is determined that such a fatal error has not occurred and communication can be accepted.

  The multifunction device 10 notifies the portable terminal 30 of the determination result indicating that communication is possible (OK) (P104). Upon receiving notification that the communication is possible from the multifunction device 10, the mobile terminal 30 selects a display mode (jam position display, skeleton display, maintenance part display, etc.) for displaying information related to the multifunction device 10 from the user. Accept and decide, and instruct the start of communication (P105). Then, it decides and notifies the MFP 10 of the display mode and communication start (P106).

  The multi-function device 10 sets the display mode notified from the mobile terminal 30 and prepares to receive image data from the mobile terminal 30 (P107). When the display mode setting and the preparation for receiving the image data are completed, the mobile terminal 30 is notified of the completion of the display mode setting and the preparation for receiving the image data (P108).

  After this notification, the portable terminal 30 receives a camera photographing operation by the user, photographs the multifunction machine 10 with the camera unit, and inputs the camera image (captured image) (P109). The input camera image of the multifunction device 10 is transmitted to the multifunction device 10 (P110).

  The multi-function device 10 stores the camera image received from the mobile terminal 30 in the RAM 13 (or nonvolatile memory 14) (P111). The multi-function device 10 analyzes the received camera image, performs processing such as feature extraction from the camera image, and specifies the shooting location of the multi-function device 10 by the mobile terminal 30. Further, the distance and angle between the camera unit of the portable terminal 30 and the shooting location of the multifunction machine 10 are calculated, the viewpoint position of the camera unit is obtained, and the three-dimensional positional relationship between the camera unit and the shooting location of the multifunction machine 10 is obtained ( P112).

  The multi-function device 10 determines display information corresponding to the shooting location of the own device from the display mode set in P107. When the display mode is skeleton display, a skeleton image having a direction corresponding to the viewpoint position of the camera unit (direction viewed from the viewpoint position) is generated, and the skeleton image is determined as display information (P113).

  The multi-function device 10 performs image composition processing, superimposes the display information on the camera image (P114), and transmits this composite image to the portable terminal 30 (P115). The portable terminal 30 displays the image (composite image) received from the multifunction machine 10 on the display unit 31 (P116).

  If the portable terminal 30 does not receive an instruction to end the operation from the user, the portable terminal 30 repeatedly performs the operations from photographing to displaying in P109 to P116 (P117). When an operation end instruction is received from the user, shooting by the camera unit and transmission of the camera image to the multifunction device 10 are stopped (P118), and the operation end is notified to the multifunction device 10 (P119).

  When receiving the notification of the operation end from the mobile terminal 30, the multi-function device 10 performs a communication end process with the mobile terminal 30 (P120). The multi-function device 10 notifies the portable terminal 30 that the end processing has been completed (P121). The portable terminal 30 ends the dedicated application program and completes all the processes (P122).

  As described above, in the MFP 10 according to the present embodiment, the photographing location of the own device is determined based on the photographing information of the own device by the portable terminal 30, and display information corresponding to the photographing location is transmitted to the portable terminal 30. Therefore, appropriate display information (display information corresponding to the shooting location) can be displayed on the mobile terminal 30 with a simple configuration without providing a large number of position detection means for detecting the position of the mobile terminal 30. Can be made.

  Further, by displaying the display information so as to be superimposed on the photographing information, it is possible to omit the creation of an image serving as a base of the multifunction machine 10 and to simplify the creation of the display information. Furthermore, it is possible to display the relationship between the actual shooting location (the actual object) shown in the shot image and the display information in an easy-to-understand manner.

  Specifically, as shown in FIG. 3, when the jam position image 50 is displayed superimposed on the photographed image, the place where the jam occurs in the actual photographed spot shown in the photographed image is displayed in an easy-to-understand manner. Thus, the user can easily grasp the jam occurrence location in the multifunction machine 10. Also, as shown in FIG. 4, when the skeleton image 60 at the jam occurrence location is combined and displayed on the photographed image, the user can easily determine the internal state of the jam occurrence location and the position of the jam paper. Can be easily grasped (it becomes easier to imagine the state of the jam occurrence location), and the removal of the jammed paper can be performed smoothly. Further, as shown in FIG. 5, when a jam occurrence location is not photographed, a jam direction image 70 indicating the direction from the photographing location to the jam occurrence location is displayed superimposed on the photographed image. The user can easily find the jam occurrence location of the multifunction machine 10.

  This makes it possible to easily find and quickly remove a large number of jammed paper that is scattered within the device and difficult to find, especially for large-scale image forming devices used for business printing. The time required can be shortened. Further, the productivity of printing can be improved by shortening the jam processing time.

  In addition to jamming, various indications corresponding to the location of the occurrence of various events such as occurrence of errors such as breakdowns, necessity of maintenance, or necessity of manual operation accompanying errors and maintenance Information (position information / skeleton image / guidance information / operation information) can be displayed on the portable terminal 30. Even when a service person handles error handling and maintenance, the position, internal state, and operation content of the corresponding location can be easily grasped from various display information displayed on the mobile terminal 30. Speed can be achieved (improvement of serviceability).

[Second Embodiment]
In the first embodiment, a case has been described in which the multifunction device 10 combines the captured image (camera image) captured by the portable terminal 30 and the display information related to the multifunction device 10. A case where the terminal 30 performs will be described.

  In the present embodiment, the function for obtaining the shooting location of the multifunction device 10 and the position of the portable terminal 30 (camera unit) based on the photographed image of the multifunction device 10 by the portable terminal 30 in the application program for the portable terminal, FIG. A function for performing the image composition processing described in P114 is incorporated. The method of specifying the shooting location based on the shot image is the same as the method performed by the multifunction machine 10 described in the first embodiment. In order to mount this function on the mobile terminal 30, the mobile terminal 30 includes collation information (image (image data) of the characteristic part of the multi-function device 10 and three-dimensional coordinate position) for specifying the shooting location of the multi-function device 10. ) Is stored. This database is the same as the database stored in the multifunction machine 10 according to the first embodiment.

  FIG. 11 is a sequence diagram illustrating a processing flow of the multifunction machine 10 and the portable terminal 30 according to the second embodiment.

  P201 to P209 in FIG. 11 are the same operations as P101 to P109 in FIG. 10 described in the first embodiment. Also, P218 to P221 in FIG. 11 are the same operations as P119 to P122 in FIG. The description of the same operation as that of the first embodiment is omitted here.

  The mobile terminal 30 analyzes a camera image (captured image) obtained by photographing the multifunction machine 10 in response to a camera photographing operation by the user, performs processing such as feature extraction from the camera image and collation with collation information, and photographing. The shooting location of the multifunction machine 10 in the image is specified by the range of the three-dimensional coordinates. The details are the same as the processing of the multifunction machine 10 in the first embodiment. In addition, the camera image is analyzed to calculate the distance and angle between the camera unit of the own device and the shooting location of the multifunction device 10 to obtain the viewpoint position of the camera unit, and the three-dimensional view of the shooting location of the camera unit and the multifunction device 10 is obtained. The positional relationship is obtained (P210).

  The mobile terminal 30 includes shooting location information (information indicating the shooting location with coordinates corresponding to a three-dimensional coordinate system preset in the multifunction device 10) indicating the shooting location of the multifunction device 10, and the viewpoint position of the camera unit. The positional relationship information indicating the three-dimensional positional relationship with the shooting location is generated and transmitted to the multifunction machine 10 (P211).

  The multifunction machine 10 identifies the shooting location of the mobile device 30 by the mobile terminal 30 from the shooting location information received from the mobile terminal 30. Further, the three-dimensional positional relationship (distance, angle) between the camera unit of the portable terminal 30 and the photographing location of the own device is recognized from the positional relationship information received from the portable terminal 30.

  The multi-function device 10 determines display information corresponding to the shooting location of the own device from the display mode set in P207. When the display mode is skeleton display, a skeleton image having a direction corresponding to the viewpoint position of the camera unit (direction seen from the viewpoint position) is generated, and the skeleton image is determined as display information (P212). The multi-function device 10 transmits this display information to the portable terminal 30 (P213).

  When receiving display information from the multifunction device 10, the portable terminal 30 performs image composition processing, overlays the display information on the camera image (P214), and displays the composite image on the display unit 31 (P215). If no instruction to end the operation is received from the user, the operations from photographing to display in P209 to P215 are repeated (P216). When receiving an instruction to end the operation from the user, shooting by the camera unit and transmission of information to the multifunction machine 10 are stopped (P217), and P218 and subsequent steps are performed.

  As described above, the mobile terminal 30 obtains the shooting location of the multifunction device 10 based on the captured image, and the multifunction device 10 obtains the shooting location information indicating the shooting location from the mobile terminal 30, thereby indicating the shooting location information. Display information corresponding to the shooting location of the own device can be transmitted to the portable terminal 30 to be displayed.

  The embodiment of the present invention has been described with reference to the drawings. However, the specific configuration is not limited to that shown in the embodiment, and the present invention can be changed or added without departing from the scope of the present invention. Included in the invention.

  For example, in the embodiment, when an event such as a jam occurs in the multi-function device 10, an example in which display information related to the event (display information related to the occurrence location of the event) is transmitted to the portable terminal 30 and displayed. However, regardless of the occurrence of such an event, display information (information such as the name, position, fluoroscopic image, and operation content of the predetermined location) corresponding to the shooting location determined based on the captured image is transmitted to the mobile terminal 30. You may make it display.

  In addition, the image forming apparatus is not limited to a multifunction machine, and may be a printer or a copier. Further, in the embodiment, various databases included in the multifunction peripheral are provided in an information processing apparatus such as a personal computer or a server, and the information processing apparatus receives a captured image of the image forming apparatus from a portable terminal, and is based on the captured image. It is also possible to determine the shooting location of the image forming apparatus and transmit display information corresponding to the shooting location to the mobile terminal for display. Further, the information processing apparatus receives from the image forming apparatus input of designated place information indicating a designated place where an event to be notified to the portable terminal is generated, and transmits display information related to the designated place to the portable terminal. It is good also as a structure to display.

10. Multifunction machine (image forming device)
11 ... CPU
12 ... ROM
13 ... RAM
DESCRIPTION OF SYMBOLS 14 ... Nonvolatile memory 15 ... Hard disk device 16 ... Display part 17 ... Operation part 18 ... Facsimile communication part 19 ... Network communication part 20 ... Scanner part 21 ... Image processing part 22 ... Printer part 23 ... Post-processing part 30 ... Portable terminal 31 ... Display unit 50 ... Jam position image 60 ... Skeleton image 70 ... Jam direction image 80 ... Mark

Claims (23)

A communication unit that communicates with a mobile terminal having a shooting function, a display function, and a communication function;
The portable terminal receives a photographed image obtained by photographing the image forming apparatus from the portable terminal, determines a photographing part photographed by the portable terminal of the image forming apparatus based on the photographed image, and determines the photographing part. A control unit for transmitting corresponding display information to the portable terminal;
An information processing apparatus comprising: The control unit determines a viewpoint position of the mobile terminal that has captured the captured image based on the captured image, and transmits display information of an orientation corresponding to the viewpoint position to the mobile terminal. Item 4. The information processing apparatus according to Item 1. The information processing apparatus according to claim 1, wherein the control unit synthesizes the display information with the captured image and transmits the synthesized information to the mobile terminal. The said control part transmits the display information according to the event which should be notified to the said portable terminal which has arisen in the said image forming apparatus, and the said imaging | photography location to the said portable terminal. The information processing apparatus according to any one of claims. An input unit for inputting designated location information indicating a designated location where an event to be notified to the mobile terminal occurs in the image forming apparatus;
When the designated location indicated by the designated location information input from the input unit is included in the shooting location, the control unit transmits first display information corresponding to the designated location to the portable terminal. And when it is not contained in the said imaging | photography location, the 2nd display information corresponding to the designated location is transmitted to the said portable terminal. The information processing apparatus of Claim 4 characterized by the above-mentioned. The information processing apparatus according to claim 5, wherein the second display information is guide information to the designated place. The display information is at least one of position information indicating a position of a predetermined portion of the image forming apparatus, a fluoroscopic image of the image forming apparatus, and operation information of the image forming apparatus. The information processing apparatus according to claim 1. The display information is information related to at least one of an error occurrence location, a maintenance location, and an operation required location in the image forming apparatus. The information processing apparatus described in 1. On the outer surface of the image forming apparatus, a predetermined mark for determining the viewpoint position of the mobile terminal that photographs the image forming apparatus is provided.
The information processing apparatus according to claim 2, wherein the control unit determines a viewpoint position of the mobile terminal based on deformation of the predetermined mark included in the captured image. The control unit receives shooting location information indicating a shooting location taken by the mobile terminal of the image forming apparatus obtained by the mobile terminal based on the captured image instead of the taken image, and The information processing apparatus according to claim 1, wherein display information corresponding to a shooting location of the image forming apparatus indicated by the shooting location information is transmitted to the portable terminal. The control unit receives viewpoint position information indicating the viewpoint position of the mobile terminal that has captured the captured image, which is obtained by the mobile terminal based on the captured image, and responds to the viewpoint position indicated by the viewpoint position information. The information processing apparatus according to claim 10, wherein display information of the orientation is transmitted to the mobile terminal. The control unit determines whether or not a location where an event to be notified to the mobile terminal occurring in the image forming apparatus is included in the shooting location, and the location where the event occurs is the shooting location When it is included in the location, the first display information is transmitted as the display information to the portable terminal, and when the location where the event occurs is not included in the shooting location, the display information is the first display information. 4. The information processing apparatus according to claim 1, wherein second display information different from the first display information is transmitted to the portable terminal. 5. The control unit determines whether or not a location where an event to be notified to the mobile terminal occurring in the image forming apparatus is included in the shooting location, and the location where the event occurs is the shooting location 4. The information processing apparatus according to claim 1, wherein a skeleton image corresponding to the shooting location is transmitted to the mobile terminal as the display information when the location is included in the location. 5. The control unit determines whether or not a location where an event to be notified to the mobile terminal occurring in the image forming apparatus is included in the shooting location, and the location where the event occurs is the shooting location The guidance information to the location where the event has occurred is transmitted to the portable terminal as the display information when it is not included in the location. Information processing device. The control unit includes, as the display information, a skeleton image corresponding to the shooting location, and guide information to a location where an event to be notified to the mobile terminal occurring in the image forming apparatus occurs. The information processing apparatus according to claim 1, wherein the information processing apparatus transmits the information to the information processing apparatus. The information processing apparatus according to any one of claims 1 to 15,
A detection unit for detecting the occurrence of an event to be notified to the mobile terminal in its own device;
Have
The said control part transmits the display information according to the event which the said detection part detected, and the said imaging | photography location to the said portable terminal. The image forming apparatus characterized by the above-mentioned. In the information processing device,
A function of communicating with a mobile terminal having a shooting function, a display function, and a communication function;
Receiving said imaged image obtained by imaging a mobile terminal Gae image forming apparatus from the portable terminal determines the imaging position taken by the mobile terminal of the image forming apparatus based on the captured image, the imaging position A function of transmitting display information according to the mobile terminal,
A program to realize In the function of transmitting display information corresponding to the shooting location to the portable terminal, whether or not the shooting location includes a location where an event to be notified to the portable terminal occurring in the image forming apparatus has occurred. When the location where the event occurs is included in the shooting location, the first display information is transmitted as the display information to the mobile terminal, and the location where the event occurs is the shooting location. 18. The program according to claim 17, wherein when it is not included, second display information different from the first display information is transmitted to the portable terminal as the display information. In the function of transmitting display information corresponding to the shooting location to the portable terminal, whether or not the shooting location includes a location where an event to be notified to the portable terminal occurring in the image forming apparatus has occurred. The skeleton image corresponding to the shooting location is transmitted to the mobile terminal as the display information when the location where the event has occurred is included in the shooting location. The listed program. In the function of transmitting display information corresponding to the shooting location to the portable terminal, whether or not the shooting location includes a location where an event to be notified to the portable terminal occurring in the image forming apparatus has occurred. If the location where the event has occurred is not included in the shooting location, guidance information to the location where the event has occurred is transmitted to the portable terminal as the display information. Item 20. The program according to Item 17 or 19. In the function of transmitting display information corresponding to the shooting location to the mobile terminal, the display information includes a skeleton image corresponding to the shooting location and an event to be notified to the mobile terminal occurring in the image forming apparatus. The program according to claim 17, wherein guidance information to the generated location is transmitted to the mobile terminal. For mobile devices with shooting, display, and communication functions,
A function for obtaining a photographing location photographed by the mobile terminal of the image forming device based on a photographed image obtained by photographing the image forming device, and transmitting photographing location information indicating the photographing location to the information processing device;
A function of receiving, from the information processing device, display information corresponding to a shooting location of the image forming apparatus indicated by the shooting location information;
A function of displaying display information received from the information processing apparatus;
A program to realize A function of obtaining a viewpoint position of the mobile terminal that has captured the captured image based on the captured image, and transmitting viewpoint position information indicating the viewpoint position to the information processing apparatus;
A function of receiving display information of an orientation corresponding to the viewpoint position of the mobile terminal indicated by the viewpoint position information from the information processing apparatus;
The program according to claim 22, wherein the program is implemented on the portable terminal.