Field Token

Field Token

The field token in the data parsing information part of the Load Definition File may contain a data token, a literal, a dynamic data item, a conditional expression, a multiple choice expression or a special token.

Data Token
Data Tokens are simple strings used to identify data fields within the Data Source File. Data within the Data Source File must be stored as token:value pairs. When the token is matched, the value portion is loaded as field data.

Example: Data Tokens

Related Data Field	Field Token	Special Processing
_PURGE_DATA	"May 31, 2004"	(d)
_ADDRESS	address
_LONGITUDE	longitude	(x4)
_LATITUDE	latitude	(y4)
_CATEGORY_ID	"64"

The data tokens in this example are address, longitude, and latitude.

Literals
Literals may be used in place of data tokens. Literals may be numeric values or date-time constants for numeric fields, or they may be text constants for string and text fields. All literals must be enclosed within a pair of quotation marks.

Example: Literals

Related Data Field	Field Token	Special Processing
_CREATE_DATE	"today"	(d)
_PURGE_DATA	"May 31, 2004"	(d)
_ADDRESS	address
_LONGITUDE	longitude	(x4)
_LATITUDE	latitude	(y4)
_CATEGORY_ID	"64"
_SOURCE	"The Daily News"

The literals "today" and "May 31, 2004" are treated as dates because the extract date (d) special processing value has been specified. The literal "today" will evaluate to the current date that this record is processed.

Dynamic Data Items
Dynamic Data Items may be used in place of data tokens. A dynamic data item may be dynamic value or a dynamic date-time value for numeric fields, or a dynamic text string for string or text fields. Relative date-time values and simple date-time expressions must be enclosed within a pair of quotation marks. All other dynamic data items must be enclosed within a pair of grave accents.

Example: Dynamic Data Items

Related Data Field	Field Token	Special Processing
_CREATE_DATE	"today"	(d)
_PURGE_DATA	"today + 7 days"	(d)
_ADDRESS	address
_LONGITUDE	longitude	(x4)
_LATITUDE	latitude	(y4)
_CATEGORY_ID	`(_FLAGS << 4 )`
_FULL_PATH	path
_CONTENT	`_@SCAN("192.168.1.2", 1048, _FULL_PATH, "BODY" )`

The first dynamic data item, "today + 7 days", evaluates to seven days after the current date. The second dynamic data item, `(__FLAGS << 4 )` shifts the bits in __FLAGS four to the left. The third dynamic item utilizes the _@SCAN function to retrieve content for indexing from the document file.

Conditional Expressions
Conditional Expressions may be used in place of data items. Conditional expressions must always be enclosed within a pair of grave accents. Any data token, literal, or dynamic data item that can be used directly in the second field can be used as the conditional result.

Example: Conditional Expressions

Related Data Field	Field Token	Special Processing
_DATE	copy_date	(d)
_PURGE_DATE	`IF (_DATE) ( _DATE + "7 days" ) ENDIF`	(d)
_ADDRESS	address

The conditional expression used in this example, checks if _DATE is defined and increments it by 7 days if it is.

Multiple Choice Expressions
Multiple Choice Expressions may be used in place of data items. Similar to conditional expressions, multiple choice expressions must always be enclosed within a pair of grave accents. Any data token, literal, or dynamic data item that can be used directly in the second field can be used as the conditional result or the alternate result.

Example: Multiple Choice Expressions

Related Data Field	Field Token	Special Processing
_CREATE_DATE	copy_date	(d)
_DATE	`IF (_CREATE_DATE) _CREATE_DATE ELSE @CDT ENDIF`	(d)
_PURGE_DATE	`( _DATE + "7 days" )`
_ADDRESS	address

The multiple choice expression used in this example, checks if _CREATE_DATE is defined and uses it if it is, if it is not, @CDT is used which evalutes to the current date and time.

Special Token
Special tokens are simple strings beginning with an at sign @, used to access meta data not stored within a document, or in one case, the entire document.

The following special tokens are supported:

Token	Description
@IDT	Returns the identity value generated for the document by the scanning software.
@CDT	Returns the current system date and time at thetime the document is being indexed. When properly controlled, the indexed date provides an ideal value for a timeline used as a primary index, since an additional scan to obtain the value is not required. Normally, a document must be partially scanned to obtain the value being used by the primary index prior to the full scan for storage and indexing of data.
@FDT	Returns the file system date and time stampassociated with the document by the operatingsystem where the document is being stored. When properly controlled, the file system date provides an ideal value for a timeline used as a primary index, since an additional scan to obtain the value is not required. Normally, a document must be partially scanned to obtain the value being used by the primary index prior to the full scan for storage and indexing of data.
@ALL	Returns the entire document for storage and indexing.
@URL	Returns the URL which may be used to access the document from the internet. The URL is generated by replacing the path to the directory tree with the base URL within the file path.
@FNM	Returns the filename for the document being processed.
@CNT	Returns the number of individual folders, beyond the root, in the path to the document being processed.
@D01	Returns the first folder, beyond the root, in the path to the document being processed.
@D02	Returns the second folder, beyond the root, in the path to the document being processed.
@D0n	Returns the nth folder, beyond the root, in the path to the document being processed.
@U01	Returns the last folder, prior to the filename, in the path to the document being processed.
@U02	Returns the next to the last folder, prior to the filename, in the path to the document being processed.
@U0n	Returns the nth folder from the last in the path to the document being processed.

Example: Special Tokens

Related Data Field	Field Token	Special Processing
_DATE	@FDT	(d)
_PURGE_DATE	"today + 7 days"	(d)
_AUTHOR	byline
_DOCUMENT	@URL
_CATEGORY_ID	"64"
_SOURCE	"The Daily News"

The special tokens used in this example are @FDT and @URL.