There are two ways create specimen records in Arctos. One way is through the data-entry application, which is a set of screens designed for specimen-by-specimen keyboarding. This application is customizable for various types of collection and has features to facilitate data-entry, such as settings that allow your entered values to be carried from a saved record to the subsequent form for a new record. The second method of data-entry is to upload the data from an externally created table in which each specimen record is one row. This method, called bulk-loading, is the method of choice for large sets of relatively homogeneous data, especially if these data are already in some digital format.
As shown above, both methods of data-entry depend on the large flattened (i.e., non-relational) table called BULKLOADER. From that table, the data are parsed into Arctos’s relational structure. Data may be entered into BULKLOADER via the Arctos Data Entry application, which may be customized for collection type and collection, by loading flat text data directly to the table, or through the use of customized imaging applications. Two applications are provided to bulkload specimen data. Once data are in table BULKLOADER they must be approved for loading by a person with manage_collection role, who takes full and final responsibility for the resultant specimen records. Various checks happen at various stages along the way, depending on how records are moved to table BULKLOADER, and errors are reported in column LOADED.
Arctos is a community which strives to record data in a meaningful fashion. Controlled data is documented in the code tables. Uncontrolled data best practices may be described in this documentation (example). You will often, especially when creating attributes, be presented with the options of leaving the attribute off or choosing either “unknown” or “not recorded.” If you know nothing, and we’ll let you, say nothing: Leave the value NULL. If the original data say nothing about hind foot length, there is no reason to create that attribute at all. However, some collections require certain attributes in the data entry screens, so you may be forced to choose a value. “unknown” is generally appropriate in this situation, but consult the code tables for actual definitions. Other data considerations are described in the concepts throughout this documentation. Search is your friend.
The person NULLing LOADED takes full responsibility for the records in the database. That, however, does not mean that the records must be perfect before being loaded. (No matter what you do, they won’t be.) The bulkloader provides limited tools with which to visualize and update data, and trying to check or fix a few thousand records in what’s essentially the worlds worst spreadsheet application (MS Excel perhaps being excepted…) is an exercise in futility. Below is outlined our approach:
Check to see if about the right number of records exist. Students have been known to both edit one record many times (creating far fewer records than should exist) and to create a new record for every “edit” (creating far more records than should exist).
Sort the data by key fields. If decimal coordinates have been entered, are they all in the correct hemisphere? Is there something reasonable in at least the ends of the taxon name field? Are there big white spaces that catch your eye when you scroll past? Are dates properly formatted? Are ID numbers properly, or at least consistently formatted?
If all that passes, go ahead and load the data. Arctos has many tools with which you can explore data, and outliers will quickly make themselves known. Once the load is complete, find the data in Arctos. (The Curatorial search pane may help.) Map everything. Return a summary, grouped by various parameters. Look at a few records in detail, and use the search screens to follow up on any anomalies that you might find. Absolute worst case scenario: Create an Encumbrance, delete the records you just loaded, and try it again. (A DBA can help you recover records which have been deleted from the bulkloader.)
To load records into Arctos proper (you’ll need role manage_collection to proceed):
- NULLify LOADED using the Browse and Edit Bulkloader application
- Wait for them to load. The load process runs hourly, but your records may be in the queue for several hours or even days. Note that LOADED is a functionally binary field: NULL==>The bulkloader may attempt to load this record; NOT NULL==>The bulkloader must ignore this record.
- Fix anything noted in LOADED.
- Enter DELETE (case-sensitive) in LOADED to get rid of a record – the record should disappear within the hour.
- Check progress under Bulkloader Status
- Fix errors and NULLify LOADED to try again
- Do NOT attempt to modify records using the edit capability of the data entry application unless those records were created using the data entry application. The data entry application is a specialized tool and is not meant for general-purpose editing.