OccuRec

 
  OCCUREC     HOW TO USE OCCUREC     NTP TIMING     READING TIMESTAMPS     INTEGRATION DETECTION     AAV FILE FORMAT     LICENSE     DOWNLOAD  

Why a new file format?


There are a number of reasons for the creation of the new file format.
  • Analogue video runs at constant frame rate - PAL (25 fps) or NTSC (29.97 fps). One of the big improvements that OccuRec brings when using integrating video cameras is saving just a single frame for a full integration interval. When the user changes the integration rate of the camera during the recording with this the frame rate will also change. AVI files only allow a constant frame rate while AAV is not dependent on a frame rate.
  • In AAV format it is possible to implement bigger compression ratio while keeping the format lossless. This is because in AAV each pixel is represented by a single byte rather than 4 bytes. This however also means that AAV is a monochrome format and the user has to choose a way of conversion to monochrome pixels. For black and white video cameras it is recommended to use the R channel and scrap the G and B channels, while for colour video cameras it is recommended to use a GrayScale conversion from (R, G, B) to Y. The user can choose the option when connecting to a capture device.
  • With the continuing advances of the analogue video camera development, more cameras begin to offer computer control. With this the settings like Gamma and Gain will be known by the driver at the time of the recording and it will be nice if we can save them as well. AVI doesn't have any place where to store non-image data such as the Gamma and Gain of the camera. AAV offers any number of additional values related to a given video frame to be also recorded in the file. Furthermore when OccuRec reads the timestamp from IOTA-VTI it also saves the retrieved timing and GPS status separately in the AAV file allowing reduction software to be able to use this directly without having to 'read' the IOTA-VTI timestamp off the video frames during the reduction.
  • In the case of an unexpected problem during the recording, such as power outage for example, the standard AVI file will likely get corrupted and may become unreadable. The new AAV format is built in a way so it can be repaired if the recording has stopped in the middle of the frame. To repair a broken AAV file use the Tools -> ADV/AAV Tools -> Repair ADV/AAV File menu item in Tangra3.


Why using ADV like file format?


Making the AAV file format very similar to the ADV format used by the Astronomical Digital Video System (ADVS) was the most natural choice for me. Already having the source code to record in ADV format and the tools (Tangra3) to read and process an ADV file it was natural to make AAV simply an ADV file for 8 bit video data. With this however the state channel and system metadata had to be modified to fit better the usage of an analogue video camera and recorder that doesn't offer for example GPS coordinates of the observation.

AAV ver.2 File Format


AAV version 2 is an extension of the ADV version 2 file format. In order to determine the AAV format version check the AAV-VERSION System Metadata (also called File Metadata) tag of the ADVv2 file. The information below describes how a processing software should handle AAV version 2 files.

This image shows a typical set of File Metadata tags recorded by the latest version of OccuRec and how to interpret the most important tags is explained below.

As AAV recordings are product of analogue video cameras there will be always a NATIVE-VIDEO-STANDARD tag. This should be used only for information purposes. The combination of NATIVE-FRAME-RATE and EFFECTIVE-FRAME-RATE tags can be used to calculate an integration rate. For example an EFFECTIVE-FRAME-RATE of 6.25 and a NATIVE-FRAME-RATE of 25.0 leads to calculating a x4 frame integration 25.0 / 6.25 = 4.00. This should be the integration rate used by the video reduction software for processing the AAV video recording.

The next two very important tags are the OSD-FIRST-LINE and OSD-LAST-LINE indicating the first and the last line (from top to bottom) where the VTI-OSD timestamp is located. As AAV files represent stacked video frames with preserved timestamps and because there is no GPS timestamps recorded, the user will need to visually read the timestamps off the screen and enter them in the video reduction software. The processing software could either split the frame into odd and even video field or could only split the even and odd lines between OSD-FIRST-LINE and OSD-LAST-LINE.

The image below shows the original OSD lines from an AAV frame and the lines when split into even and odd parts.



By definition the timestamps in an AAV frame will be of the first and the last video field inside the stacked frame from an integration interval. In the example above with a 25 fps PAL camera 19:37:52.9522 is the end timestamp of the first video field and 19:37:53.0522 is the end timestamp of the last video fields. We know this because it is evident that IOTA-VTI has been used for timestamping. If a different video time inserter is used then the interpretation of the timestamp from the first and last video fields could be different. The actual interpretation is something for the end user to figure. In our case above the exposure started at 19:37:52.9322 and finished at 19:37:53.0522, which means it took 120 ms, which corresponds exactly to 4 PAL frames. The video reduction software should ask the user to enter a timestamp to be used for the integrated 120 ms frame and may guide the user in this process.

It is also important to note that by design the very fist and very last frame inside an AAV file should always use no integration. Depending on how the file may have been additionally chopped however, this may not always be the case. The purpose of the no-integration frames is to determine the order of video fields inside the frame. Usually the top (odd) field should be the first one and should have the smaller timestamp. However some frame grabbers could have a defect and could combine video fields incorrectly. If this is the case then calculations from the split timestamps above may show that the frame duration was different than the expected one from the integration rate. This could be because the timestamp of the very first and very last video fields in the integration period may not correspond to the actual chronological first and last video fields. In those very rare cases the non-integrated frames can help identify this problem. For such videos there will be a timestamp bias of one video field.

Individual AAV frames may have the following status channel values:

Tag Description
SystemTime The computer system time at the time when the frame was saved in the AAV file.
IntegratedFrames The number of frames counted in the current integration period. Please note that occasionally it is possible based on the system load that some frames may be dropped or may be OccuRec will not detect correctly the integration boundaries for some integrated frames. In such cases the number of IntegratedFrames recorded here may be different than the integration rate calculated from the EFFECTIVE-FRAME-RATE and the NATIVE-FRAME-RATE file metadata.
StartFrame The internal recording software frame id of the first frame included in the current integration period.
EndFrame The internal recording software frame id of the last frame included in the current integration period.
Gain The camera Gain setting.
Gamma The camera Gamma setting.
CameraExposure The camera Exposure setting.
NTPStartTimestamp An NTP-based timestamp associated with the first frame in the integration period. This is based on OccuRec's own NTP timestamping, rather than the PC clock's NTP synced clock.
NTPEndTimestamp An NTP-based timestamp associated with the last frame in the integration period. This is based on OccuRec's own NTP timestamping, rather than the PC clock's NTP synced clock.
NTPTimestampError Maximum error of the NTP timestamp in milliseconds (based on PING time to the NTP servers).


It is also worth noting that:

  • OccuRec will continue to save the AAV16-NORMAL file metadata value for each file even that the same value is also recorded in the ADVv2 MaxPixelValue internal value. A video reduction software should prefer the ADVv2's internal MaxPixelValue value and should ignore the AAV16-NORMAL file metadata value.
  • The CALIBRATION stream of an AAV file will typically contain up to 25 sequential non-integrated frames with a FRAME-TYPE status channel value of 'VTI-OSD-CALIBRATION'. These frames may be used for example for training an OCR algorithm as it will be guaranteed that they are sequential with no gaps.
  • A very small number of AAV files may actually have an ADVv2 UTC timestamp. This will be based on an on-the-fly OCR of the IOTA-VTI timestamp processed during the recording.
  • OccuRec has a special recording mode where no video data may actually be recorded. These are not occultation videos and can be identified by a file metadata tag called STATUS-CHANNEL-ONLY having a value of 1




What are the values recorded by OccuRec in the Status Channel of the AAV (ver.1) file?


Field Name Description Notes
StartFrame The frame id of the first frame included in the current integration period.  
EndFrame The frame id of the last frame included in the current integration period.  
IntegratedFrames The number of frames counted in the current integration period.  
SystemTime The computer system time at the time when the frame was saved in the AAV file.  
StartFrameTimestamp String representation of the VTI OSD of the start video field as recognized by the OCR engine. Only present if the OCR has been configured and is working.
EndFrameTimestamp String representation of the VTI OSD of the end video field as recognized by the OCR engine. Only present if the OCR has been configured and is working.
GPSTrackedSatellites The number of tracked satellites at the time of the VTI timing. Only present if the OCR has been configured and is working.
GPSAlmanacStatus The alamanc status at the time of the VTI timing. 0 = Waiting, 1 = Good. Only present if the OCR has been configured and is working.
GPSFixStatus The GPS Fix at the time of the VTI timing. 0 = Unknown, 1 = Internal Timekeeping, 2 = G Fix, 3 = P Fix. Only present if the OCR has been configured and is working.