Storage and analysis techniques for fast 2-D camera data on NSTX

 

W. M. Davisa*, D.M. Mastrovitoa, C.E. Bushb, D.A. Gatesa, R.J. Maquedac, J.E. Menarda, N. Nishinod,

P.G. Roneya, A.L. Roquemorea, S. Sabbaghe,  B.C. Strattona, S.J. Zwebena

 

aPrinceton Plasma Physics Laboratory, P.O. Box 451, Princeton, NJ, 08543, USA

bOak Ridge National Laboratory, Oak Ridge TN, 37831, USA

c Nova Photonics, Princeton, NJ, 08543, USA

d Hiroshima University, Hiroshima, Japan

eColumbia University, New York, NY 10027, USA

 

 

 

 

 

Abstract

 

Animations from fast 2-D camera data are facilitating the investigation of spatially distributed phenomena in high-temperature plasmas. The National Spherical Torus Experiment (NSTX) now has six fast camera systems, and more are expected to be added. Image capture rates vary between 1000 and 500,000 frames/second. Archiving and retrieving this data is a challenge for data repositories and networks. For example, if all camera data had been archived during the 2004 run, the total amount of data from NSTX (300MB per pulse) would have doubled, and this year, one new camera alone can acquire 2GB per pulse. The paper will describe the storage strategies, and compare some data compression techniques used for NSTX. Tools which animate camera data, synchronized with displays from other plasma diagnostics and simulations, allow scientists to gain insights and observe correlations that would be difficult with conventional tools. A labor-saving  technique is described for archiving fast camera images from a vendor’s system into MDSplus and to AVI files. Examples of specific analysis and display techniques are presented. Future challenges are also discussed.

 

Keywords: NSTX, MDSplus, Fast Cameras, Data Compression, Plasma Images

1. Introduction

 

The National Spherical Torus Experiment (NSTX) began fusion experiments at the Princeton Plasma Physics Laboratory (PPPL) in 1999 [1].  During an experimental cycle, or “shot,” a plasma is produced for about one second and about 400 megabytes (uncompressed) of raw data is acquired from instruments in dozens of subsystems, hosted by Unix, VMS and Windows computers. This raw data is, for the most part, transferred to and stored on centralized data servers and is available within a few minutes to anyone “subscribed” to particular signals. Some of this raw data is also input to analysis programs which

Fig. 1. Data from three cameras combined with an EFIT reconstruction [10] and plots of other plasma conditions vs. time. As time progresses in the composed animation,  the vertical dashed line on the plasma-conditions vs. time plot moves to the right, and individual frames from each camera and the flux contours from EFIT, are displayed at the appropriate time.

 

Fig. 2. 9 msec exposure from a Phantom 7 Camera showing a MARFE on the center stack and fast filamentation on the edge (128x128 pixels, contrast enhanced).

store their results in similar structures, or in databases, both to provide immediate feedback to the machine operators and diagnostic physicists so they may make adjustments for the next shot, and to provide a repository of information for later off-line analysis. A typical NSTX run day produces about 40 shots. NSTX runs for between 60 and 80 days a year. 2.7 terabytes of raw and analyzed NSTX data (compressed) currently reside on disk. 2-D camera data is accounting for an increasing proportion of the total data on NSTX. Two-dimensional camera data is a valuable tool for studying turbulence, gas injection, ion transport, pellet injection, wall interaction, and other phenomena which affect energy confinement and heating efficiency in fusion energy production.

Most raw and analyzed data from NSTX, including raw camera images, is stored in MDSplus [2,3] but animation files are stored elsewhere currently. Better compression techniques are being investigated since our high-storage, high-availability disk space currently costs over $8 (US) per gigabyte, and networks and tape backups add to the cost.

 

Table 1

2-D camera capabilities of selected NSTX diagnostics

 

 

Diagnostic

Max

Fps

# of frames

Typical

Res.

#

bits

Total

MB

RF Antenna

5K

300

512x240

14

36

Soft X-ray

500K

300

64x64

14

3.6

Divertor

40K

8192

64x64

8

32

Phantom 7

150K

175K

64x64

12

2000

 

 

 

2. 2-D Camera Data Storage Requirements

 

Table 1 shows examples of 4 of the 6 fast camera diagnostics in use on NSTX. Some cameras are used for a visual inspection of large-scale features, such as gas injection and surface heating. For such purposes 30-1000 frames per second (fps) can be adequate. Longer exposure times, such as those for the large image in Fig. 1, blur small-scale, quickly changing phenomena, such as the filaments shown in the 9 microsecond exposure from the Phantom 7 camera in Fig. 2

The faster, higher-resolution cameras place a large burden on the data archiving system. The

Fig. 3.  Event-Driven Fast Camera Data Archiving.

Phantom-7 camera [4] used for, among other things, the Gas Puff Imaging diagnostic, can produce 2GB of data per shot, more than all the other NSTX diagnostics combined. Currently, a diagnostician monitors the data between shots and manually saves sequences of interest in the central repository. Any other data of potential interest is saved directly to DVDs. Similarly, the Divertor Camera (the small images in Fig. 1) [5] save most data only on the local PC.

 

3. Archiving 2-D Camera Data into MDSplus and AVI files

 

NSTX 2-D Camera Diagnostics use several ways to store their images in a central repository. Most write the frames into MDSplus, either as 2-D individual frames, or as 3-D arrays. Most camera diagnostics arrive at NSTX as “turn-key” systems, with control code written in LabView, Java, or C. When the source code is available, “hooks” may be added to write directly into MDSplus. However, in most cases it is easier and more maintainable to use a system that is independent of a vendor’s software.

One such system is illustrated in Fig. 3. A hardware start trigger (usually from a CAMAC module set through MDSplus) is received by vendor-provided camera and PC system (1). The vendor software writes individual TIFF files to a directory on the PC whose name is set to the NSTX shot number initially, and incremented for every shot. A PPPL-written program installed on the PC, PCDA_plus, waits for an MDSplus event known to occur after all the frames have been written, “ACQ_DONE” in this example (2), and initiates a secure copy (scp), using a public-domain program called pscp.exe [6] (3). Previously generated Kerberos [7] tickets allow this transfer to occur securely without passwords needing to be entered each time or being sent in clear text. When the transfer is completed, the MDSplus event “MOVED” is declared (4). Another program running on the Linux server (or anywhere with an MDSplus connection) waits for the “MOVED” event and writes all the frames into MDSplus as one 3-D signal (5), and declares the event “LOADED” when finished (6). Another program on the AVI PC waits for this event, reads the signal from MDSplus (7) and creates an animation file in a web-accessible directory (8). This program uses public-domain IDL code and DLLs [8] to create AVI (Audio-Video Interleave) files with a choice of compression-decompression methods (codecs). Anyone can then play these animations in a browser, download them to their favorite movie player, or send them to the NSTX Display Wall (9). The codec used for compression must also be on the play back computer.

 

Table 2

Compression ratios for two types of 2-D image filesa using different methods

 

 

Method

 

Plasma Only

 

Complex Graphsb

GIF

3.5

4.0

JPEG, Qual=100

4.6

2.5

PNG

6.5

6.7

JPEG Qual=75

28.2

7.7

JPEG Qual=25

39.1

14.2

 

a8-bit images

bThe Complex graphs contain lines and text as well as plasma images, as shown in Fig.1.

 

4. Compression techniques

 

MDSplus has an option for writing data in compressed format, but there is only one compression method for all data types. For our 8-bit integer image data, we typically see a compression ratio of 1.3-2.0 using MDSplus compression. Standard image compression methods vary substantially depending on the data (see Table 2), but usually do much better. PNG (Portable Network Graphics) gives the best compression ratio without loss of data quality.

Table 3

Compression ratios for two types of animation filesa using different compression methods (codecs)

 

 

Codec used

 

Plasma Only

 

Complex Graphs

Cinepak by Radius

11

32

mSoft Video 1

22

14

MPEG-1

41

89

Indeo Video 5.10

64

104

XviDMPEG-4

184

170

VP31 Compressor

384

70

 

a24-bit images

 

Many images of plasmas are non-distinct and can tolerate the lower-quality JPEG compressions and still show the relevant features. Since video compression methods can compress in time, as well as within a frame, they can achieve even higher compression ratios. Table 3 shows some results of 24-bit animations from NSTX for various codecs. Clearly, for optimal storage, the results of various compression methods should be examined for each type of image.

 

5. Analysis techniques

 

Image analysis techniques of NSTX plasmas usually consist of visual examination and manual correlation with other phenomena, such as MHD activity, the onset of H-mode, structure in light emission signals, etc. [9]. Composing synchronized animations from camera images, EFIT reconstructions [10], and time histories, as shown in Fig. 1, facilitate making visual correlations.  Some quantitative studies have measured the size and motion of regions of enhanced local line emission near the plasma edge, commonly referred to as “blobs,” and compared them to other diagnostics and to theoretical models [11]. More of such work is needed in order to utilize the increasingly large amounts of camera data.

 

6. Future Challenges

 

Fast 2-D camera data will continue to make up an increasing portion of the data acquisition load on NSTX and other machines. Saving this data will be challenging, especially for long pulse and continuous-operation machines. We need to determine which compression techniques are best suited for various diagnostics and what the acceptable trade-offs are between detail losses and storage costs, if any.

It is convenient to access one- and two-dimensional signals stored in MDSplus with general tools that do not have to be custom made, but few generalized tools for MDSplus deal with animations. Some lossless compression occurs when images are stored in MDSplus (a factor of 1.3-2), but built-in methods for images would save much more space.

Although storage requirements can be met given sufficient expenditure, effectively analyzing huge amounts of data is still problem. As the operator of the Gas Puff Imaging Diagnostic said of the data shown in Fig. 2 “almost every sequence of 300 frames within the 45000 has something interesting ...and different from the previous 300 frames sequence.” [12] Techniques are needed to categorize image sequences automatically, and to store and retrieve them in databases. Image databases exist [13], and descriptive keywords can be entered by hand, but automatic discovery techniques for plasma phenomena would be valuable.

 

7. Summary

 

Fast 2-D camera diagnostics are important for understanding important phenomena in plasmas, and are becoming increasingly prevalent on NSTX. Storing and analyzing the data is challenging, both in terms of labor (programming and analyzing) and in terms of storage media. Compression methods help, but should be applied intelligently for the proper trade off between resolution loss, cost and accessibility. Automated analysis methods are only beginning to aid NSTX physicists.

 

Acknowledgements

 

This work was supported by DOE Contract DE-AC02-76CH03073.

 

References

 

[1] S. Kaye et al., The Physics Design of the National Spherical Torus Experiment. Fusion Technology, 36, July 1999, p. 16, or http://nstx.pppl.gov/

 

 [2] J.A. Stillerman, T.W. Fredian, K.A. Klare, G. Manduchi, MDSplus Data Acquisition System. Review of Scientific Instruments 68 (1) January 1997, p. 939. http://www.mdsplus.org/

 

[3] W. Davis, P. Roney, T. Carroll, T. Gibney, D. Mastrovito, The Use of MDSplus on NSTX at PPPL, Fusion Eng. Des. 60 (2002), 247-251.

 

[4] Phantom 7 Fast Camera, http://www.scitech.com.au/cameras/highspeed/pdf/visionres/Phantomv7.0spec&info.pdf

 

[5] N. Nishino, e, Results with the NSTX Fast Divertor Camera,  J. of Plasma and Fus. Research 78 (2002)

[6] Open ssh for Windows, http://www.openssh.com/windows.html

 

[7] Kerberos, http://web.mit.edu/kerberos/www/.

 

[8] IDL to AVI code for PCs http://www.rsinc.com/codebank/search.asp?FID=139

 

[9] R. MAINGI, et al., H-mode turbulence, power threshold ELM and pedestal studies in NSTX, submitted to Nucl. Fusion, 2005.

 

[10] S.A. Sabbagh, S.M. Kaye, J. Menard, et al., Equilibrium Properties of Spherical Torus Plasmas in NSTX,  Nucl. Fusion 41 (2001) 1601.

 

[11] S. Zweben, et al.,  High-speed imaging of edge turbulence in NSTX, Nucl. Fusion 44 (2004) 134-153

 

[12] e-mail correspondence from R. Maqueda to W. Davis June 30, 2005.

 

[13] Almagest OpenSource software http://www.princeton.edu/~almagest/opensource/

 

 


* Corresponding author. E-mail: bdavis@pppl.gov,  Phone: (609)-243-2546,  Fax: (609)-243-3086