A Guide to Getting the Best Performance with Large Datasets

Are Large Datasets Slowing You Down?

Careful pre-planning and a strategic computer configuration help maximize productivity while using Trimble Business Center (TBC) and Trimble RealWorks.

Data processing and management presents new challenges to surveyors as multiple data collection tools, such as high-resolution cameras, high-speed scanners, and mobile mapping systems, generate larger volumes of data than ever before. Projects that contain data from these sources can exceed hundreds of Gigabytes in size and easily overwhelm computing and storage capabilities. Completing a project efficiently requires making data easily accessible to users when they need it and building hardware in an optimal configuration to make the most of the software functionality. By following a few guidelines, even very large datasets can be collected, processed, stored and managed in a workable manner.

Here are a few key points:

• Consider the end deliverable and select the appropriate point density and/or resolution needed to produce the necessary information. Only collect as much data as you need to get the job done.
• Assess how many people will need access to the data and where they are located. Within one office, portable hard drives can be the most efficient method to transfer data. If data is being shared between multiple locations, high-speed network connections to local servers are preferable if available.
• Create procedures for naming/versioning copies of the data that are generated during processing. This will avoid confusion and make it easier for multiple people to work on different parts of the project, which will ultimately be combined in the final deliverable.
• Determine by what method the client wants to receive the deliverable. This can range from Dropbox to shipping hard drives to a web-based viewer such as Trimble Clarity.
• Use local storage for ongoing work and archive finished projects for the long term. Archive data should include information that might be needed for follow-on work or to answer questions, but not redundant edited versions that could be quickly re-created from the raw data.
• Evaluate the pros and cons of the three approaches to storage and processing ─ local PC, local network server, or cloud ─ and how each relates to your situation. There are tradeoffs between speed, cost, flexibility, scalability, and security that will influence your choices and ensure you are able to meet your objectives.
• Recognize that software products deliver the best results on computers with a balanced harmonious configuration. For example, if you have slow drives and a fast CPU, or fast RAM and a slow CPU, there will be performance bottlenecks that have nothing to do with the software.
• Keep in mind that for Trimble scanning software, the speed of the CPU, disk drives and RAM have a bigger impact on the overall speed of processing than the GPU speed. Be aware of the distinction between manufacturers’ “Pro” products versus their “Gaming” products when selecting data drives. The Gaming products tend to have faster cores but fewer lanes, which equates to slower RAID operations. Overall, data drive speed is determined equally by the number of lanes, the type of lanes, and the drive speed, so use a holistic approach to make your selection.

By considering the above topics and designing your workflow and computer network in an optimal manner, your scanning software will deliver the best results possible, saving you time on every project. For more detailed information about any of these topics, download the whitepaper A Guide to Getting the Best Performance with Large Datasets.