Automated LiDAR Extraction Software Requested by Chris Thornton, Division of Right of Way and Land Surveys, Caltrans Developed by the Advanced Highway Maintenance and Construction Technology (AHMCT) Research Center October 18, 2021 The Caltrans Division of Research, Innovation and System Information (DRISI) receives and evaluates numerous research problem statements for funding every year. DRISI conducts Preliminary Investigations on these problem statements to better scope and prioritize the proposed research in light of existing credible work on the topics nationally and internationally. Online and print sources for Preliminary Investigations include the National Cooperative Highway Research Program (NCHRP) and other Transportation Research Board (TRB) programs, the American Association of State Highway and Transportation Officials (AASHTO), the research and practices of other transportation agencies, and related academic and industry research. The views and conclusions in cited works, while generally peer reviewed or published by authoritative sources, may not be accepted without qualification by all experts in the field. The contents of this document reflect the views of the authors, who are responsible for the facts and accuracy of the data presented herein. The contents do not necessarily reflect the official views or policies of the California Department of Transportation (Caltrans), the State of California, or the Federal Highway Administration. This document does not constitute a standard, specification, or regulation. No part of this publication should be construed as an endorsement for a commercial product, manufacturer, contractor, or consultant. Any trade names or photos of commercial products appearing in this publication are for clarity only. Table of Contents Executive Summary Page 3 Background Page 3 Summary of Findings Page 3 Gaps in Findings Page 5 Next Steps Page 6 Detailed Findings Page 8 Background Page 8 Caltrans Survey Requirements and Background Page 8 Summary of DOTs and Consultants Survey Results Page 8 Literature Review of LiDAR Feature Extraction Software Page 16 COTS Software Page 18 Open-Source Software Page 29 Related Research and Resources Page 30 Contacts Page 33 State Agencies Page 33 Survey Services Consultants Page 36 References Page 37 Appendix A: Caltrans Feature Extraction Requirements Page 38 Line Features Page 38 Point Features Page 42 Appendix B: Automated LiDAR Point Cloud Feature Extraction Software Questionnaire Page 45 Executive Summary Background The California Department of Transportation (DOT) (Caltrans) Surveys Program currently uses Certainty 3D TopoDOT as its Light Detection and Ranging (LiDAR) extraction software. Surveys need to utilize the most accurate and automated software available for feature extraction. Currently feature extraction from LiDAR data is a labor-intensive and time-consuming process. Feature extraction is broken into two categories: manual and semi-automated. The user viewing LiDAR point clouds uses images from still cameras to assist in extraction process. There have been significant developments in recent years in Artificial Intelligence (AI), Machine Learning (ML), and related fields. LiDAR software would benefit dramatically from the application of these technologies (ML and AI), which could greatly reduce the user burden for LiDAR data feature extraction. Any commercial off-the-shelf (COTS) software packages that applies AI or ML to this task could greatly benefit Caltrans. Availability of such tools would be crucial to support the latest mandates for asset management as well as Building Information Management (BIM) for Infrastructure. For example, the Federal Highway Administration (FHWA) is pushing state DOTs to adopt All Road Network of Linear Referenced Data (ARNOLD) and to collect Model Inventory of Roadway Elements (MIRE 2.0) for their safety program. The Caltrans Division of Maintenance will be required to keep better inventory of existing sign and roadside features. Automation is vital to reduce cost and time for roadway and roadside asset feature extraction from point cloud data gathered using Mobile Terrestrial Laser Scanning (MTLS). The Caltrans preliminary investigation (PI) entitled Statewide Asset Data Collection and Management: Survey of Practice provided a comprehensive literature review of the best practices of current statewide Asset Data Collection and Management. This report provides crucial background information for prioritizing asset features required for DOT business functions and processes across entire DOTs. It referenced a thorough list of fundamental state DOT, FHWA, National Cooperative Highway Research Program (NCHRP), and Transportation Research Board (TRB) documents concerning statewide asset data collection and management. This PI reviews the leading LiDAR software tools related to feature extraction and assesses their capabilities with respect to application of AI, ML, and similar advanced technology. This may include assessment of the software’s ability to learn from operator decisions as well as to automatically extract and classify features. The PI summarizes the state of COTS software packages for this essential feature. Summary of Findings Summary of DOTs and Consultants Survey Results A survey questionnaire was sent to state DOTs and private contractors to determine their: - Awareness of availability of COTS software packages with advanced feature extraction capabilities - Experiences in use of COTS software packages with advanced feature extraction capabilities - Mechanisms used to fund acquisition of COTS software packages with advanced feature extraction capabilities - Benefits from the use of COTS software packages with advanced feature extraction capabilities for that DOT The survey questionnaire was sent out in a Microsoft (MS) Form format and a (Portable Document Format [PDF] version attached in Appendix B for reference), allowing the respondents to answer the questionnaire online in an effort to minimize response burden. The raw data from this survey questionnaire is available in MS Excel format upon request. Twenty-five responses were received. Nineteen respondents are state DOTs, and six respondents are private consultants from four companies. Two of the DOTs responded that they do not perform LiDAR point cloud feature extraction. Nine out of seventeen DOTs (53%) use TopoDOT for their primary point cloud feature extraction software. Two DOTs use the Mandli Roadview software for feature extraction. The remaining DOTs used one of the following software packages: Trimble Business Center, Terrasolid, PolyWorks, and Cardinal System VR mapping. Nine out of seventeen (53%) DOTs use more than one feature extraction software. Table 1 shows feature extraction software used by respondents’ organizations by asset feature type. One common reason for using more than one software for feature extraction is that users find different software have strengths and weaknesses in their available extraction tools. Ten out of seventeen DOTs (59%) use overhead funds for their software acquisition. Six out of seventeen DOTs (35%) use project funds for software acquisition. Only two DOTs (12%) out of seventeen responded DOTs have documented the benefits (e.g., efficiency improvement, time savings, labor savings, etc.) of their purchased COTS software. Commercial off-the-shelf (COTS) Software The literature review and assessment of COTS software is based upon information from the vendors, including marketing materials, websites, online forums, YouTube videos, online webinars, and specification sheets or documents. The review covers semiautomated and automated feature extraction for: - Roadway assets, including lane lines and other striping, manholes, drainage, concrete barriers, median signage, dikes, edges of pavement, curbs, and gutters, etc. - Roadside assets, such as poles, trees, signs, guardrail, etc. - Structures - Overhead lines - Point cloud classifications/segmentations. Point cloud classification or segmentation does not yield the final desired 3D modeling lines and points data directly. However, the classification or segmentation results can speed up subsequent feature extraction processes. A literature review was performed on ten COTS feature extraction software for their extraction automation. The ten software are Atlas Survey Control Centre, Bentley Descartes, Certainty 3D TopoDOT, ESRI ArcGIS Suite, EZDataMD LidarTools, InnovMetric PolyWorks Surveyor, Leica Cyclone suite, Rapidlasso LASTools, Terrasolid Suite, and Trimble Business Center. Table 2 summarizes the results of the COTS software feature extraction tools automation review. ESRI ArcGIS Suite, Trimble Business Center, and EZDataMD LidarTools employed ML or similar technologies in their point cloud classification/segmentation tools. The ESRI ArcGIS Python Application Programming Interface (API) enables users to train and develop their own deep learning neural network for point cloud segmentation and classification. However, only a power line and pole ML-powered classification tool is available as an ArcGIS extension for public usage. More pre-trained ML classifications and extraction tools may be available from ESRI or the user community. TerraScan, a product from the Terrasolid suite, supports custom macros for point cloud classifications. Most of the automatic classification routines can be combined in macros for batch processing. Cyclone 3DR has scripting support and has developed two custom scripts for curb and gutter as well as power line extraction. Many of the reviewed software products have semi-automated tools for breaklines, curb, and gutter extraction. Some of these tools may also work on concrete barriers and guardrails. Furthermore, few of the COTS products have automation with human quality assurance/quality control (QA/QC) tools for power line, tree, pole, and sign extraction. Their degree of automation may vary, especially in power line extraction. Gaps in Findings Academic research and COTS software have shown good results when applying ML on point cloud classification segmentations. However, the current challenge is to produce three-dimensional (3D) lines and points from the resulting classified or segmented point cloud through the application of other algorithms, such as cluster, centroid calculation, shape fitting, and/or maximum or minimum height. These results may be used to produce the 3D computer-aided design (CAD) or geographic information system (GIS) data for final deliverables. ESRI ArcGIS provides a Python API for users to conduct their own custom ML training for their specific dataset. Creating a sample dataset for ML training can be quite laborintensive and time-consuming. The process requires users with strong ML and deep learning knowledge and Python programming proficiency. In addition, ML training is computationally expensive. General users will have to wait for others to integrate ML powered feature extraction tools into COTS software. Few software programs have demonstrated automated sign extraction tools with good results. Most California signs contain a significant amount of text. In order to extract text from signs, a user will need to manually input text attributes from the sign images. Challenges of adopting new software - Workflows and data transfer - Caltrans has developed extensive and detailed Caltrans Survey Asset Collection (CSAC) conventional survey workflow and MTLS workflow to produce deliverables in Civil 3D drawing format. TBC software is integrated into the CSAC survey workflow, and TopoDOT is integrated into the existing MTLS workflow. The customized workflows were developed in both TBC and MicroStation to ensure seamless and lossless data transfer to Civil 3D. Adopting any new feature extraction software would require development of a workflow to ensure extracted digital data is transferred to Caltrans-specific Civil 3D format without errors or data loss. - Added overhead in software training cost - personnel time, development of training materials and workflows - Added time and labor overhead in data transfer/import/export between software - Additional licensing costs - Additional Information Technology costs such as software installation and maintenance - Additional costs in data storage for temporary native data files - Users will have to maintain multiple formats for the same data while performing data extraction. Next Steps - Follow-up with DOT survey respondents - Develop a DOT peer-to-peer support group for point cloud data to exchange best practices, information, and ideas - Conduct a survey of Caltrans’ feature extraction software users to rank assets or features that are the most time consuming or labor intensive to extract - Software evaluation and testing. The software should be evaluated based on the following performance metrics: - Extraction accuracy: position accuracy, false positive and false negative rates - Feature attributes extraction automation -Processing speed (human time and computer time) -Data import and export overhead time -Time for navigation in the point cloud to see and identify the assets -Time or number of mouse clicks for extracting the assets using the software tools -Time to QA/QC the assets, visual aids such as multiple views of the asset points, “local magnifying view” on a separate window that reduces navigation time in zoom in and zoom out -Effectiveness and robustness of the extraction tools such as how well they may follow a curve or missing points due to blocked laser line of sight by debris or vegetation. Some ground classification tools would filter out curb and concrete barriers. o Time for subsequent line editing. The software tool produced lines may not be 100% accurate. Detailed Findings Background Caltrans Surveys utilizes Certainty 3D TopoDOT as its primary extraction software. Current processes are both laborious and time-consuming. There have been significant developments in AI, ML, and related fields resulting in significantly advanced LiDAR software that would benefit from application of these techniques and could greatly reduce the user burden for LiDAR data feature extraction, e.g., of pavement markings, barriers, signs, and similar highway infrastructure. Any COTS software packages that apply advanced techniques, such as AI or ML, to this task could greatly benefit Caltrans. Availability of such tools is crucial to support the latest mandates for asset management as well as for survey-grade projects as performed by Caltrans Surveys and to minimize project costs. The Caltrans Division of Right of Way and Land Surveys needs an up-to-date understanding of the availability of advanced technologies in LiDAR software packages, particularly for feature extraction. This PI reviews the leading LiDAR software tools related to feature extraction and assesses their capabilities with respect to application of AI, ML, and similar advanced techniques. This may include assessment of the software’s ability to learn from operator decisions as well as to automatically extract and classify features. The PI summarizes the state of COTS software packages for this essential feature. Caltrans Survey Requirements and Background Caltrans has owned and operated fixed and mobile terrestrial laser scanning systems for over nine years. Caltrans has used InnovMetric PolyWorks Surveyor, Virtual Geomatics, Leica Cyclone, and Trimble RealWorks. Caltrans Photogrammetry uses Terrasolid Suite including TerraScan and Cardinal System VR mapping software. For MTLS collected point clouds, Certainty 3D TopoDOT is the primary feature extraction tool used by Caltrans Surveys. Appendix A provides a table of point and line features that may be required in a Caltrans survey project. In addition, Caltrans has developed and documented customized workflows and tools for survey data created in MicroStation or Trimble Business Center (TBC) to Autodesk Civil 3D. For example, a custom Caltrans MicroStation workspace and cell library was created to standardize survey products. Caltrans Office of Land Surveys (OLS) has developed a custom TBC feature code library called Caltrans Survey Asset Collection (CSAC). Beyond point cloud feature extraction tools, other custom survey business automation tools and software are vital to integrate survey data to existing software and workflows to final survey deliverables. Summary of DOTs and Consultants Survey Results A survey questionnaire was sent to state DOTs and their contractors to determine their: - Awareness of availability of COTS software packages with advanced feature extraction capabilities - Experience using COTS software packages with advanced feature extraction capabilities - Mechanisms used to fund acquisition of COTS software packages with advanced feature extraction capabilities - Benefits from the use of COTS software packages with advanced feature extraction capabilities for that DOT The survey questionnaire is in an MS Form format (PDF version attached in Appendix B), allowing the respondents to answer the questionnaire online in an effort to minimize response burden. The raw data is available in MS Excel format upon request. Survey of Practices Results Twenty-five persons responded to the survey questionnaire. The raw survey results are provided in MS Excel format. Nineteen respondents were state DOT personnel, and six respondents were private consultants from four different companies. Two (West Virginia and New Hampshire) of the 19 state DOTs responded that they did not perform LiDAR point cloud feature extraction. Seventeen state DOTs respondents performed LiDAR point cloud feature extraction. Figure 1: Survey questionnaire respondents’ organization distribution 6 Private Consultants 19 D.O.T State Personnel Consultants Figure 1: Survey questionnaire respondents’ organization distribution The survey asked respondents, “What primary software is used for point cloud feature extraction?” Figure 2 shows the frequency count and distribution of different software used by DOT and contractor respondents. The data collected shows TopoDOT is the primary software preferred for feature extraction. Figure 2: Primary point cloud feature extraction software used 0 1 2 1 0 1 9 1 2 1 0 0 0 1 1 2 1 0 0 2 4 6 8 10 12 Atlas SCC Cardinal Systems VR Mapping Mandli Roadview Polyworks Survey Control Centre (SCC) TerraSolid / TerraScan TopoDOT Trimble Business Center Various Multiple Software 2 State D.O.Ts State DOTs Consultants Figure 2: Primary point cloud feature extraction software used The survey asked respondents, “Are different software packages used to extract specific features?” and “What software is being used to extract the following features: Curb & Gutter, Roadway Asset, Roadside Assets, Overhead Lines, Structures, Point Cloud Classifications.” Fourteen respondents out of the combined total 23 respondents, and 9 out of 17 DOT respondents, use more than one feature extraction software. Table 1, on the following page, shows feature extraction software used by respondents’ organizations sorted by asset feature type. An asterisk (*) after the software name indicates that it is the primary software used by the organization. Table 1: Point cloud feature extraction software used by respondents’ organizations by asset feature type. * Primary software, Survey Control Centre (SCC), and Trimble Business Center (TBC) Agency Additional Software Curb & Gutter Roadway Asset Roadside Assets Overhead Lines Structures Point Cloud Classification Jacobs Yes SCC* SCC* SCC* SCC* SCC* SCC* Honey Badger Analytics Yes TopoDOT* TopoDOT* TopoDOT* TopoDOT* TBC ArcGIS Pro Seiler Yes TBC* TBC* TBC* TBC* TBC* TBC* DataDEV Yes Terrasolid* Trimble RealWorks Terrasolid* Terrasolid* Terrasolid* Terrasolid* Jacobs Yes TopoDOT TopoDOT TopoDOT TopoDOT & Pointerra TopoDOT LAStools, TopoDOT, or Pointerra Jacobs No Atlas SCC* Atlas SCC* Atlas SCC* Atlas SCC* Atlas SCC* Atlas SCC* Montana DOT Yes Cardinal Systems VR* Cardinal Systems VR* Cardinal Systems VR* TBC Hawaii DOT Yes Mandli Roadview* Mandli Roadview* Mandli Roadview* Mandli Roadview* Mandli Roadview * Mandli Roadview* Michigan DOT No TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* Maine DOT No TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* NDDOT Yes TerraScan* TopoDOT TerraScan* TerraScan* TerraScan * TerraScan* NCDOT No Various Various Various Various Various Various Nevada DOT No TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* SCDOT No TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* WSDOT Yes PolyWorks* PolyWorks* PolyWorks* PolyWorks* TopoDOT NJDOT No TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* Indiana DOT Yes TBC* TBC* TBC* TBC* TBC* TBC* NYSDOT Yes TopoDOT* TopoDOT* TopoDOT* TopoDOT* Leica Cyclone Leica Cyclone Caltrans No TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* IL DOT No TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* TopoDOT* MnDOT Yes TopoDOT TopoDOT Other TopoDOT Ohio DOT Yes TopoDOT* TopoDOT* TopoDOT* TopoDOT* TerraScan TerraScan Oregon DOT Yes TopoDOT* TopoDOT* & Bentley Descartes TopoDOT* & Bentley Descartes TopoDOT* TopoDOT* TerraScan, Leica Pegasus, & TopoDOT* The reasons for using more than one feature extraction software were given by some respondents and are listed below: From consultants: - “Atlas Survey Control Center (SCC) is our standard regional survey processing platform which has high-end point cloud handling and extraction capabilities which surpass standard packages like LSS. SCC point cloud module has an engine which can sufficiently handle large point cloud data sets for topographical extraction.” - “Usability of format for end use customer.” - “Different platforms provide different solutions and depending on the client, market or output required we choose the best tool.” From DOTs - “Trimble Business Center does the best job for Point Cloud Classifications. Cardinal Systems VR MAPPING SOFTWARE works better for mapping. Overhead Lines are the only thing I have found to be more efficient using automated feature extraction. Due to the amount of editing needed for curb and gutter automated feature extraction I have found manual extraction to be more efficient at this time.” - "Most of our data is mobile terrestrial LiDAR data." - “NDDOT has utilized TopoDOT for surface creation in the past. We just obtained TerraScan/TerraModeler software suite and were hoping to utilize the combination of the two to featurize curb and gutter, along with other linear features.” - “Every software has strength and weakness. We use the appropriate software for what we believe is the better and faster way to extract features and do analysis.” - “We use whatever software tool works the best for the extraction needed.” - “We feel certain packages work better for certain tasks. Some areas don't have access to all the software choices and some are specialized per office depending on preference or time spent doing extraction.” - “Established workflows and better results.” - “We use different applications because they all perform specific tasks better than the other application. For example, TerraScan does a much better job with point cloud classification than TopoDOT. When extracting assets, the application depends on the size of the project: if it is just a small project, then Descartes works well, but if it is larger, then we'll use TopoDOT.” The survey asked respondents, “Does your organization have any standards and/or best practices for features extraction?” Eleven out of twenty-three respondents and six out of seventeen DOT respondents have some standards and/or best practices for feature extraction. Some of the respondents’ organizations have their standards and/or best practices documents available as listed below: - Illinois DOT’s standards and/or best practices document available via email request (see contacts). - Ohio DOT document, which lists all Standard Mapping Collection Features from page 28 to page 30 (https://www.dot.state.oh.us/Divisions/Engineering/CaddMapping/Survey%20an d%20Mapping%20Specifications/Survey%20and%20Mapping%20Specifications% 20January%2017%202020.pdf) - American Society for Photogrammetry and Remote Sensing (ASPRS) Positional Accuracy Standards (https://www.asprs.org/a/society/committees/standards/Positional_Accuracy_St andards.pdf) - Minnesota DOT Mapping Manual (http://www.dot.state.mn.us/surveying/pdf/mappingmanual.pdf) Only two DOTs out of all twenty-three respondents are interested in other COTS software packages with advanced feature extraction capabilities for improved efficiency: - Hawaii DOT is interested in Allvision (https://www.allvision.io/2021/03/09/newoffering/) - Minnesota DOT is interested in TopoDOT, GeoCue, and ESRI suite Five out of twenty-three total respondents and four out of seventeen DOT respondents have evaluated or have experience using other COTS software packages with advanced feature extraction capabilities. Their descriptions of their experiences are summarized below. - “Trimble Business Center, Leica Cyclone, and Cardinal Systems VR MAPPING SOFTWARE all have the same issues when using advanced feature extraction. If it is not a perfect surface then they will produce anomalies that require manual editing. Since we are most likely surveying roadway features that are older with vegetation and soil covering edges of concrete sidewalk and top back of curb. Manual mapping of these edges allow for manipulation of horizontal and vertical vertices. This process I have found to be more accurate and efficient due to the amount of editing needed after automated line work is generated. Also it can be difficult to detect errors after automated processes another reason it seems to make sense to get it right the first time.” - “We host our LiDAR data on the cloud now with all the extracted features.” - “We use all said packages but the key word is advanced. All those packages are not fully automated and still require manual input.” - “Some packages were not user friendly and duplicated what TopoDOT, TBC, or TerraScan can do.” Figure 3 and Figure 4 show the respondent organizations’ funding mechanisms for acquisition of feature extraction software packages. The “other” category in Figure 3 consisted of a consultant using a "sale” as a fund source and the Michigan DOT using combined grant, project, and overhead funds for their software acquisition. The majority of the DOTs (59%) use overhead funds for software acquisition. Figure 3: Funding mechanisms of all respondents’ organizations for the acquisition of feature extraction software 1 Capital Expenditure 8 Project funds 11 Overhead funds 1 Cost Recovery from hourly rate 2 Other Figure 3: Funding mechanisms of all respondents’ organizations for the acquisition of feature extraction software Figure 4: DOT funding mechanisms for the acquisition of feature extraction software 6 Project funds 10 Overhead funds 1 Grants, project & overhead funds Figure 4: DOT funding mechanisms for the acquisition of feature extraction software The survey asked respondents, “Have you documented the benefits (e.g. efficiency improvement, time savings, labor savings, etc.) of your COTS software package?” Two out of seventeen (12%) DOTs and three out of four consultant organizations have documented the benefits (e.g., efficiency improvement, time savings, labor savings, etc.) of their purchased COTS software. In addition, three consultant organizations out of four have documented the benefits. For the question, “Please provide any other information, documents, or feedback that you believe may be of value for this research. If you feel your agency has more information to offer on this subject, please provide contact information to allow for follow-up discussion,” the following survey responses were received. From consultants: - “Extraction is largely a manual process undertaken by our internal survey teams with little support or provision from GID, as they confirm they do not have the current skill base or capacity to support processing of these mass datasets. We are looking for that step up in automation to help speed up our extraction processes whilst maintaining the level of quality and completeness our clients expect in terms of topographical outputs.” From DOTs: - “The link to the recording of my 3D presentation of 26APR16 can be found on the following page: EDC-3 3D Engineered Models Webinar Series (https://www.fhwa.dot.gov/construction/3d/edc3webinars.cfm)” “It’s similar to a bit of what is covered in this HEEP presentation: Hawaii Dept. of Transportation: Photolog to Digilog: 3D Asset Management (http://www.heep.org/conference/archive/2014/files/presentations/photolog_to _digilog.pdf) and includes a bit about our work in developing some DTM models in the Virtual Reality Design Studio environment developed by forum8: State of Hawaii, Department of Transportation, Integration of Point Cloud Data and Photo-Log Images of Highways within VR-Design Studio (https://www.forum8.co.jp/user/user107-e.htm)" - “For terrestrial scanning, we rely on Cyclone to process the data and register it. We then export it to TopoDOT for manual/semi-automated feature extraction. For our UAS LiDAR, we use the RIEGL software products to process the data, then export it to TopoDOT also.” - “NDDOT would like to see the overall results of the survey.” - “SCDOT uses TopoDOT for feature extraction. Recent project used the TopoDOT auto cross slope tool to generate reports for our Construction office. Our Consultants use other feature extraction software in addition to TopoDOT.” - “We own several point clouds software. Cyclone is only for point cloud alignment, ClearEdge3D is for metal structures mapping, TopoDOT will become a major player after we migrate to Open Road Design (64bit environment) and PolyWorks will still be used for features extraction and fixing misalignment in point clouds.” - “Mobile LiDAR was added to our pavement analysis collection contract in 2017. LiDAR data was collected and extracted in 2017 & 2018 giving Illinois a complete statewide coverage. We are still finding uses for the data and working it into our processes.” Literature Review of LiDAR Feature Extraction Software The literature review and assessment of COTS software is based on information from the vendors, e.g., marketing materials, websites, online forums, YouTube videos, online webinars, and specification sheets or documents. This review and assessment did not include any evaluation or testing of the software packages. Furthermore, this review does not cover other point cloud software functions, such as registration, data management, data distribution/publishing/streaming, and extraction for other industries such as railway, archeology, and building and plant modeling. Manual editing or labor-intensive extraction tools are not considered. The assessment covers only semiautomated to automated feature extraction for: - Roadway assets, including lane lines and other striping, manholes, drainage, concrete barriers, median signage, dikes, edge of pavement, curb, and gutter, etc. - Roadside assets such as poles, trees, signs, guardrail, etc. - Structures - Overhead lines - Point cloud classifications/segmentations Automation Definitions Different vendors or users define the term “automate” differently. In this document, the author generally classifies an extraction tool degree of automation as manual, semiautomatic, automatic with human QA/QC, and automatic. - The extraction tool can be considered “manual” if the users must click on almost every point on an extracted line to create a line. Some tools assist the user in picking the points without zoom in and zoom out or picking the highest or lower points. In this case, the users still must click on all or many of the points to create a line. - The extraction tool can be considered “semi-automatic” if the users must click on one to four points in order create a line or few lines at the same time. - The extraction tool can be considered “automatic with human QA/QC” if the tool extracts multiple features or assets with attributes with few user inputs. However, the user is required to review the results or enter additional asset attributes not detected by software. For example, tree, sign, and pole extraction tools are automatic with human QA/QC. Users are often required to review the extracted data and add attribute information. - The extraction tool can be considered “automatic” if the tool can execute the process with little or no user input. In some cases, the tool may be started in batch mode and run in the background. Generally, point cloud classification or segmentation tools are considered automatic in this document. Point cloud classification or segmentation does not yield the final desired 3D modeling lines and point data directly. The classification or segmentation results can speed up subsequent feature extraction processes. However, for some software tools, it is difficult to classify the degree of automation. Some tools require extensive setup before extracting the feature automatically. Finally, some of these tools may or may not use ML or AI technology since the software makers often do not reveal technologies or algorithms used by the software tool. Software improvements are made frequently. This review may soon become outdated as this industry changes quickly. COTS Software Atlas Survey Control Centre (SCC) Point Cloud Module SCC – Point Cloud (http://atlascomputers.ie/scc-point-cloud/) Atlas SCC can import, visualize, analyze, and create high quality point cloud models and drawings from point clouds. A series of training videos is available on YouTube at SCC automated kerb tracing (https://youtu.be/AM8rH7IprIU). Detailed instructions of their feature extraction tool are also available at the Atlas SCC forum on Laser Scanning Forum (https://laserscanningforum.com/). Laser Scanning Forum requires user registration login. In addition, the Atlas SCC Point Cloud Module manual is available online at SCC Point Cloud Tutorials (http://www.atlas-files.com/scc-users/SCC_PointCloudModule.pdf). The latest version Atlas SCC Point Cloud Module is 13.31.3, and its release notes are available online at SCC 13-31-3 Release Notes (http://atlas-files.com/sccusers/ SCC%2013-31-3%20release%20notes.pdf). Atlas includes the following point cloud feature extraction tools: - Extract outlines of points with a similar color or intensity, such as road markings or shop signage - Semi-automatic tool to extract linear features, such as curb, gutter and barriers using an alignment and moving section template. See video: SCC - Adding road markings to a point cloud (https://youtu.be/MzWRel8-YqU) and SCC - Modelling mobile LIDAR data (https://youtu.be/pH8coHQgFoY) - Semi-automatic pothole extraction tool is also available. See Automated pothole extraction (https://laserscanningforum.com/forum/viewtopic.php?f=220&t=16194) for more detail. - Automatic filtering of vegetation, cars, street furniture, and noise to create a clean ground model. This tool classifies points in the point cloud automatically. See video: SCC automated kerb tracing (https://youtu.be/AM8rH7IprIU) Bentley Descartes Descartes - Advanced Processing for 3D Imagery (https:/www.bentley.com/en/products/product-line/reality-modeling-software/bentleydescartes) Descartes is Bentley’s reality modeling software. It is designed to display very large scalable terrain models and has a variety of modes, such as smooth shading with shadows, aspect angle, elevation, slope, and contours. It can be used to extract break lines, paint lines, surfaces, planes, cylinders, and cylinder center lines from reality models. It has the following point cloud feature semi-automatic extraction tools applicable to DOT needs: - The 3D Line Following tool can be used to extract 3D lines semi-automatically, such as lane lines and power lines, directly from one or many point clouds. The user specifies a point on the line to extract and a direction to follow. Detailed written instructions and a video demonstration are available at the following links: Descartes SS4 - Point Clouds - 3D Line following to extract road paint lines (https://www.youtube.com/watch?v=aXx0lIMPXL4) and 3D Line Following (https://communities.bentley.com/products/3d_imaging_and_point_cloud_softw are/w/wiki/6994/3d-line-following) - The smart snap feature is a manual tool for extracting breaklines. Bentley Descartes’ online instructions on breakline extraction are provided in the link below. Using the Point Cloud Tools (https://docs.bentley.com/LiveContent/web/Descartes%20CONNECT%20Editionv7/ en/GUID-FCBB34E4-093E-FBA6-29BF-4871D956135C.html) - Users may also use the cross-section tool to create a breakline manually. The Bentley Descartes CONNECT Edition: Extract Breaklines (https://youtu.be/fzp31xKKYUk) YouTube video provides more details. Certainty 3D TopoDOT Certainty 3D TopoDOT (https:/new.certainty3d.com/) works in conjunction with Bentley MicroStation software. TopoDOT University (http://certainty3d.com/university/) and TopoDOT wiki (http://www.certainty3d.com/wiki/index.php) require register/login and have an extensive detailed user manual, tutorials, and training resources for TopoDOT users. In addition, the TopoDOT YouTube channel (https://www.youtube.com/channel/UCJ1nTk3o2H4l9e_eaGeru9w) provides users with new information and new tools and software features. The Caltrans Mobile Terrestrial Laser Scanning Guidelines has documented detailed usage for several tools commonly used in Caltrans feature exaction. Some of these tools have been updated by TopoDOT. The TopoDOT Extraction Tools video (https://youtu.be/R_oyXpSnDSI) shows the use of many of the feature extraction tools listed below. Semi-automatic tools are: - Extraction by intensity for lane lines - Drape element to data - Breakline extraction for curbs, gutters, concrete barriers, and guardrails - Power line clearance - Elevation grid - Cross-section by points - Road condition report tool analysis, including pavement distress, ruts, and potholes Automatic with human QA/QC tools are: - Auto asset identification - Classify ground The latest TopoDOT 2020.1 webinar video (https://youtu.be/0W7pshxJVNM) shows a few software tools that improve the speed of feature extraction and reduce labor. - Power line extraction, the new Wire Extraction by Direction is a semi-automatic tool that can be used to extract power lines. The new Power Line Extraction by Cross-Section tool enable users to extract multiple wires at one time and perform human QA/QC and manual editing afterwards. The TopoDOT Wire Extraction by Direction video (https://youtu.be/uMXAu_ZV0Mk) demonstrates the use of these tools. - The new Template by XSection, a semi-automatic tool detects lines extract curbs, gutters, and concrete barriers with less user input - Place Smart-Cell tool employs a template to extract ramps, such as American Disabilities Act (ADA) ramps, with few manual adjustments. See the video Auto Fit clip (https://youtu.be/y1egZpohKwc) for details. - The Flat Asset tool is a semi-automatic tool that can be used to extract painted markings on the ground, such as arrows, STOP text, etc., as show in this video Flat Asset Demo (https://youtu.be/PAmAq61qxMQ) - The Pedestrian Crossing extraction tool extracts crosswalks on the ground semiautomatically Generally, TopoDOT develops and names their tools based on the intended features to be extracted by the tool. ESRI ArcGIS Desktop and ArcGIS Pro ESRI ArcGIS Desktop and ArcGIS Pro (https://www.esri.com/en-us/home) software is a powerful Geographic Information System (GIS) software suite. Caltrans has licenses for ArcMap and ArcGIS Pro. Both ArcGIS Desktop and ArcGIS Pro 2.8 support point cloud data and have point cloud analysis tools. ArcGIS Pro allows users to use statistical or machine learning classification methods to classify point clouds. Deep learning is a type of machine learning that relies on multiple layers of nonlinear processing for feature identification and pattern recognition described in Introduction to deep learning and point clouds (https://pro.arcgis.com/en/pro-app/latest/help/data/lasdataset/ introduction-to-deep-learning-and-point-clouds.htm). Users may use ArcGIS API for Python (https://developers.arcgis.com/python/) to develop and train their own ML models. The user may develop a custom ArcGIS extension combining the ML model Point Cloud Segmentation using PointCNN (https://developers.arcgis.com/python/guide/point-cloud-segmentation-usingpointcnn/) and Python code and made available in ArcGIS for other users. In addition, user community developers often provide their ArcGIS extension to other ArcGIS users. ESRI has provided some examples of applying ML/AI technologies on point cloud classification and feature detection/extraction on their website: Extracting Value from Point Clouds using ArcGIS (https://storymaps.arcgis.com/stories/adfc1dbc4fee4e6b9de21e298d48d61b). Their examples include power lines and pole extraction, building footprints extraction, and Object extraction from Mobile LiDAR point clouds with Machine Learning (https://medium.com/geoai/object-extraction-from-mobile-lidar-point-clouds-withmachine- learning-cb15fcbb5597). Dmitry Kudinov, Senior Principal Data Scientist at ESRI, demonstrated application of ML technologies in point cloud segmentation and subsequent feature extraction in the video AI for LiDAR Feature Extraction (https://www.esri.com/videos/watch?videoid=M3znR7_HiUQ). ArcGIS supports deep neural network Convolutional Neural Network (CNN) and Point Convolutional Neural Network (PointCNN), which allow for efficient point cloud classification/segmentation. Specifically, at the video time 35:50, the presenter shows the result of applied PointCNN ML for MTLS point cloud classification. PointCNN is for feature learning from point clouds presented by Li et al. in the PointCNN: Convolution On X-Transformed Points research article published in 2018 [1]. CNNs are the convolution operator leverage spatially-local correlation in data represented densely in grids such as images. CNN is commonly applied for object recognition and segmentation on images. However, point clouds are irregular and unordered. Therefore, CNN cannot be directly applied against features associated with the points in the point cloud. Li et al. proposed to learn an X-transformation from the input points to simultaneously promote two causes: the first is the weighting of the input features associated with the points, and the second is the permutation of the points into a latent and potentially canonical order. Element-wise product and sum operations of the typical convolution operator are subsequently applied on the X-transformed features. Their experiments show that PointCNN achieves on par or better performance than state-of-the-art methods. After the point cloud classification, traditional algorithms (density-based clustering) and statistical methods are used to extract trees and traffic lights positions. However, with the exception of the power line classification tool shown below, the other ML/AI classifications and extractions demonstrated by Dmitry Kudinov were not found on the web for general users. An extensive ArcGIS extension search is needed to determine if there are other user community developed ML/AI powered feature extraction tools. Some ArcGIS extension tools may be run in the background in batch models. - Power line Classification ArcGIS 3D Analyst extension for ArcGIS Pro Power Line Classification (https://www.arcgis.com/home/item.html?id=6ce6dae2d62c4037afc3a3abd19a fb11) is a deep learning model that can classify point clouds into distribution wires, poles, or background. It utilizes a PointCNN model and the ArcGIS API for Python. This tool was developed using aerial LiDAR point cloud data. The article “PointCNN: Replacing 50,000 man-hours with AI” claimed that 50,000 man-hours are saved. 1 1 PointCNN: replacing 50,000 man hours with AI (https://medium.com/geoai/pointcnnreplacing- 50-000-man-hours-with-ai-d7397c1e7ffe) The GeoAI Hub (https://landing-geoai.hub.arcgis.com/) provides links to webinars and articles on the state-of-the-art application of ML and AI technologies on point cloud processing. In addition, the ESRI website, Search Results for "lidar feature extraction" (https://www.esri.com/videos/search?text=lidar%20feature%20extraction), also provides webinars on LiDAR feature extraction. In addition, ArcGIS has automated tools to create digital terrain models (DTM) and digital surface models (DSM) from point clouds as well as point cloud classification tools using traditional algorithms. EZDataMD LidarTools LidarTools: Automated Lidar Data Management and Processing (https://lidartools.com/) Erzhuo Che and Michael J. Olsen at Oregon State University have been developing new methods to improve feature extraction from point clouds. They have developed a Mo-norvana segmentation method to automatically classify a point cloud [2]–[6]. They presented their research result at the TRB Geospatial Data Program Meeting in August 2020 (https://trb.secure-platform.com/a/page/Geospatial/GeospatialDataProgram). Their presentation is available at Efficient Point Cloud Segmentation of Transportation Assets for Improving the Workability of Lidar Data (https://cdn.filestackcontent.com/qxqg6da4T4ugo9sCdWW7?EZRA%20Che%20and%20 Michael%20Olson%2020200805_LidarTools_Submitted.pdf). See also Presentation video for Efficient point cloud segmentation of transportation assets (https://vimeo.com/449411214). Their research resulted in a software called EZDataMD LidarTools. An AHMCT researcher has provided sample mobile scan data of California roadway for testing and evaluation. The EZDataMD LidarTools automatically segment/classify roadway feature points into unique class (group). Figure 5 and Figure 6 show the results. . The automated segmentation/classification process supports subsequent feature extraction or modeling. The users can isolate each feature group of points from other points in the point cloud to aid the feature extraction. However, the current version of EZDataMD LidarTools does not automatically extract the outline of a feature group of points nor extract the center point of any pole-like feature. Figure 5: EZData LidarTool segmentation results on sample mobile scan point cloud of a California freeway. Each color represents a unique class. For example, the roadway ground points are in one class (color), and the curb face is in another class. This figure shows each bridge column is in a unique class. Figure 5: EZData LidarTool segmentation results on sample mobile scan point cloud of a California freeway. Each color represents a unique class. For example, the roadway ground points are in one class (color), and the curb face is in another class. This figure shows each bridge column is in a unique class. Figure 6: EZData LidarTool segmentation results on sample mobile scan point cloud of a California roadway. Each color represents a unique class. For example, the roadway ground points are in one class (color), and the curb face is in another class. Figure 6: EZData LidarTool segmentation results on sample mobile scan point cloud of a California roadway. Each color represents a unique class. For example, the roadway ground points are in one class (color), and the curb face is in another class. InnovMetric PolyWorks Surveyor InnovMetric (https://www.innovmetric.com/) Caltrans used InnovMetric PolyWorks Surveyor Version 11 and 12 for feature extraction from 2012 to 2014. It has two semi-automatic feature extraction tools commonly used to extract DOT features. - One tool follows and extracts lines based on color or intensity value, and it is commonly used to extract lane lines. - Another tool tracks sharp edges, step edges, and other geometry to extract lines. It is very effective in extracting curbs, gutters, bridge columns, guardrails, and concrete barriers. However, no current information on PolyWorks Surveyor software can be found from the Innovmetric.com website. PolyWorks Surveyor is part of the PolyWorks suite designed for surveyor applications. The current PolyWorks suite focuses on inspection, 3D analysis, quality control, and reverse engineering. Leica Cyclone Suite Leica Cyclone (https://leica-geosystems.com/en-us/products/laserscanners/ software/leica-cyclone) Cyclone suite supports Leica Geosystems laser scanner field data collection operations, scan registration, data quality assurance and quality control (QA/QC), feature extraction and modeling for topographic surveys and BIM models, movies/animations, and 3D data distribution and management. Cyclone survey and Cyclone 3D Reshaper (3DR) modules have point cloud feature extraction tools. Cyclone Survey - Cyclone Virtual Surveyor and SmartPicks tools are often used in conjunction to assist and improve user feature extraction speed and reduce labor. (See YouTube video Topographic Workflows Part 3 Virtual Surveyor 102 Linework (https://youtu.be/JTB_DCeb7-Y) for detail). The SmartPicks tool helps users to select the local highest, lowest, or average point in a small local area. A user still has to click on every single point in the point cloud to extract lines or points, making this a manual extraction tool with computer assistance. - At laserscanningforum.com an interview with a Cyclone user can be found that describes how the breakline generation can be semi-automated. There are “sharp edge,” "fit edge," and “edge-pick” selections that help a user extract breaklines semi-automatically to automatically. However, detailed information on the breakline automation tool was not found on publicly available websites. - The Point on Grid tool is useful for creating a DSM and DTM automatically (see video Topographic Workflows Part 4 Points on a Grid [https://youtu.be/orDyRVBZqRo] for detail). Cyclone 3DR Cyclone 3DR has a set of flexible meshing tools and a rich scripting engine that enables feature extraction automation. Some of the semi-automated and automated tools, developed for building 3D modeling, rail industry, tunnel analysis, etc., may be useful or applicable for DOT feature extraction. A Cyclone tutorial video may be found at Cyclone tutorial (https://leica-geosystems.com/products/laser-scanners/software/leicacyclone/ leica-cyclone-3dr). - The Curb Extraction tutorial video on the 3DR tutorial webpage (video 7/15 on Leica Cyclone 3DR - All-in-one Deliverable Solution [https://leicageosystems. com/products/laser-scanners/software/leica-cyclone/leica-cyclone- 3dr]) shows the use of their custom curb script for extracting curbs and gutters from a point cloud. The process may be considered semi-automatic to automatic with human QA/QC. The video also shows (see Figure 8, top left side) an Electric Line Extraction script icon, which suggests that Cyclone 3DR can extract power lines/overhead lines semi-automatically or automatically with human QA/QC. However, no video or webpage was found to show how the Electric Line Extraction script can be used. Figure 7: Cyclone 3DR tutorial video screenshot showing curb extraction and electric line extraction custom scripts icons (on the top left side) Figure 7: Cyclone 3DR tutorial video screenshot showing curb extraction and electric line extraction custom scripts icons (on the top left side) - The two YouTube video web links below show an automatic with human QA/QC process for breakline extraction. 3DReshaper: massive Breaking Lines extraction (https://youtu.be/vgXAM8I2Q4k) and Tips & Tricks #51 - Extract breaking lines with slope analysis (https://youtu.be/pNtbnSyY7wM) - Cyclone 3DR has an automatic ground extraction tool with noise removal for contour and DSMDTM creations. In this context, “noise” refers to points collected from moving objects, such as automobiles and humans, during laser scanning data collection. In the case of DOT laser scanning surveys, the “noise” from traveling vehicles on the roadway can hinder feature extraction by human and software automation. - Cyclone 3DR has an edge following tool developed for building modeling/architecture/heritage preservation applications. It may be applicable for DOT feature extraction. There is very limited detailed information on Cyclone feature extraction tools publicly available online, which limited independent verification of the software. There is some uncertainty on the degree of automation of the aforementioned tools. In addition, new scripts may be made available to better automate the extraction process for other DOT features. Rapidlasso LASTools LAStools (https://rapidlasso.com/lastools/) The LAStools software suite is a collection of batch-scriptable and multicore command line tools. LAStools can also be run via a native GUI in ArcGIS versions 9.3 and higher and QGIS versions 1.8 and higher. LAStools has two classification tools (lasground and lasclassify) for point clouds. These two command line tools can perform the process automatically in the background or on a server with no user interaction. - Lasground – extracts bare-earth and classifies points into ground points (class = 2) and non-ground points (class = 1) in a point cloud. - Lasclassify – classifies buildings and high vegetation (e.g., trees) in a point cloud. This tool requires that the bare-earth points have already been classified and the elevation of each point above the ground was already computed with the Lasheight tool in LAStools. Terrasolid Suite Terrasolid (https://terrasolid.com/) TerraScan software works in conjunction with Bentley MicroStation. Users may view the Terrasolid suite latest webinar recordings and pre-recorded training sessions after registration. TerraScan provides tools to create 3D vector models, feature extractions, point cloud visualizations, etc. The TerraScan manual is available in PDF at TerraScan user guide (https://terrasolid.com/guides/tscan.pdf). Moreover, the Terrasolid YouTube channel (https://www.youtube.com/c/Terrasolid/videos) provides instructional videos. TerraScan supports custom macro for point cloud classifications. Most of the automatic classification routines can be combined in macros for batch processing. In combination with batch processing, macros can be executed outside TerraScan and MicroStation. Their tutorial video shows that the features are often extracted after classifications. - The Classify Groups Tutorial video (https://youtu.be/PAM_VrZrWNY) shows the point cloud classification process including computation of distance above ground, group assignment steps, and group classification process. - The Find Poles Using TerraScan Object Detection video (https://youtu.be/f4oR41Flw0Y) shows users may extract poles from the point cloud. The process requires some amount of setup before the pole extraction and may be considered semi-automatic to automatic with human QA/QC depending on the object library setup status. The steps are as follows: 1. Pre-classify the point cloud and assign groups 2. Define sample objects 3. Create cell elements for mapping poles 4. Add new object items to the TerraScan Object Library 5. Find poles automatically - The Detect Wires Automatically Using Detect Wires video (https://youtu.be/2_ZC1lWlMPw) shows how users can extract power lines from a point cloud. The process requires some amount of setup before extraction and may be considered semi-automatic. The steps are: 1. Place a tower string line 2. Classify points on wires of a power line 3. Create line string elements in the CAD file for all detected wires automatically - TerraScan provides tools for analyzing road surface distress, such as potholes and rutting, using semi-automatic extraction. The following two YouTube videos show details of the process: - Find Poles Using TerraScan Object Detection (https://youtu.be/NDWeUrTgJWU) Results of TerraScan Tools Used to Analyze and Design Road Surface Improvements (https://youtu.be/rjRIYqPEJ44) Trimble Business Center (TBC) Point Cloud Module Version 5.51 Trimble Business Center (https://geospatial.trimble.com/products-and-solutions/trimblebusiness- center) The TBC Point Cloud module was designed for point cloud extraction for large geographical area point clouds. Presentation file TBC resources (https://drive.google.com/file/d/1s8vUvTkuv8EAjXto3d2deRpqwV5mgGXj/view?usp=sh aring) has links to various tutorials and a YouTube channel to learn more about TBC and its modules. Both the TBC Survey and Construction YouTube channel (https://www.youtube.com/user/TBCSurvey) and the Trimble Mobile Mapping YouTube channel (https://www.youtube.com/c/TrimbleMobileMapping) have webinar recordings and tutorial videos showing point cloud feature extraction in detail using different point cloud datasets. The TBC point cloud extraction tools incorporate advanced ML technology developed for Trimble eCognition software. - The TBC point cloud classification tool has different options for indoor or outdoor point clouds. The indoor point classifications are walls, floor, grated floor, and celling. Most DOT point clouds are outdoors. The TBC classification tool can classify outdoor points into building, ground, high vegetation, poles and signs, and power lines classes as shown in Figure 8. The process is automated with little user input. The user can reclassify previously classified points with different options. However, classification does not provide the final extracted features that the DOT requires. Classifying points does improve the subsequent feature extraction process by increasing feature recognition, which leads to increases in extraction speed and accuracy. The user can choose to apply the extraction tool on only one class of points. Figure 8: Example TBC classification results. Different colors represent different point classification. Brown is ground, blue is building, red is poles and signs, and dark green is high vegetation. Figure 8: Example TBC classification results. Different colors represent different point classification. Brown is ground, blue is building, red is poles and signs, and dark green is high vegetation. - The TBC linear feature extraction tool includes power lines and curb and gutter extraction. Both tools are semi-automatic requiring few clicks on the point cloud. The curb and gutter extraction tool will likely work on concrete barriers and possibly sound walls; however, no demonstrations exist for these features. Details of both tools are provided in the following videos. The power lines tool may also be used to extract lane lines. - Essentials: MX9 Data Processing and Deliverables in TBC demonstration/instruction video (https://youtu.be/hFVZY1J3_Io) -TBC Overhead Lines and Curb + Gutter Linear Feature Extraction demonstration/ instruction video (https://youtu.be/Vk1YrVLY6O8) - The TBC Trees, Poles, and Signs extraction tools are automated with human QA/QC. The YouTube video Essentials: MX9 Feature Extraction in TBC (ttps://youtu.be/H6JCc1CNFv8) demonstrated how the automatic Trees, Poles, and Signs extraction tool is used. The tool also has a manual option. The automated tool guides the user to perform QA/QC on each feature detected/extracted. The tree extraction tool also estimates the dimensions of tree spread, trunk diameter, and tree height. The Signs and Poles extraction tool also estimates diameter and height. The user can manually enter other attribute information during the QA/QC process. Table 2: COTS Software Feature Extraction Tools Automation Summary Software Name Roadway assets Roadside assets Structure Overhead Lines Classification / Segmentation Atlas SCC ** Bentley Descartes ** TopoDOT ** to *** *** ** *** **** ESRI suite *** *** **** EZData LidarTools **** PolyWorks ** ** Leica Cyclone suite ** to *** * ** Rapidlasso LASTools **** Terrasolid suite ** *** *** **** TBC ** *** *** **** * = Manuel Extraction ** = Semi-Automatic Extraction *** = Automatic with Human QA/QC **** = Automatic Extraction OPEN-SOURCE SOFTWARE CloudCompare CloudCompare (https://www.danielgm.net/cc/) CloudCompare is a free, open-source 3D point cloud and triangular mesh processing software. It was originally designed to perform comparisons between two dense 3D point clouds or between a point cloud and a triangular mesh. In subsequent versions, it was enhanced with generic point cloud processing software, including many advanced algorithms, registration, resampling, color/normal/scalar fields handling, statistics computation, sensor management, interactive or automatic segmentation, display enhancement, etc. It allows for plugins developed by other researchers. The following plugins do not utilize ML technologies, but they may be applicable for DOT feature extraction: - The CSF plugin, an automatic point cloud classification tool, Airborne LiDAR Data Filtering Algorithm Based on Cloth Simulation (https://www.cloudcompare.org/doc/wiki/index.php?title=CSF_(plugin)), uses Cloth Simulation Filter (CSF) to extract and classify ground points in discrete return LiDAR point clouds. - The Virtual Broom plugin (https://www.cloudcompare.org/doc/wiki/index.php?title=Virtual_broom_(plugi n)), a manual tool, deletes/segments LiDAR “traffic noise” from the point cloud. “Traffic noise” comes from scan points of moving traffic during data collection. - RANSAC Shape Detection plugin (https://www.cloudcompare.org/doc/wiki/index.php?title=RANSAC_Shape_Det ection_(plugin)) detects and fits plane, sphere, cylinder, cone, and torus to selected point clouds. Point Data Abstraction Library (PDAL) PDAL (https://pdal.io/about.html#) is a C++ library for translating and manipulating point cloud data. It is very much like the GDAL library which handles raster and vector data. PDAL provides a list of command line-base software tools. In particular, the PDAL ground command (https://pdal.io/apps/ground.html) can be used to segment the input point cloud into ground versus non-ground returns using PDAL filters.smrf and filters.outlier functions. In other words, PDAL ground command can automatically classify ground points in a point cloud in batch process by a computer in the office or server room. Related Research and Resources PI-0234 - Statewide Asset Data Collection and Management: Survey of Practice, Caltrans, March 2020 (https://dot.ca.gov/-/media/dot-media/programs/researchinnovation- system-information/documents/preliminary-investigations/pi-0234aa11y. pdf) This recent PI on asset data collection includes a comprehensive literature review of the state-of-practice of current statewide asset data collection and management. This report provides crucial background information for prioritizing asset features required for DOT business functions and processes across the entire DOT. It referenced a thorough list of fundamental state DOT, FHWA, NCHRP, and TRB documents that include statewide asset data collection and management. Online Resources Guidelines for the Use of Mobile LiDAR in Transportation E-Learning website (http://learnmobilelidar.com/) This website is designed to facilitate the interactive learning of the Mobile LiDAR Guidelines document and serve as a central hub for discussion and transmission of knowledge among the mobile LIDAR community. It provides links to key mobile LiDAR reference documents for MTLS practitioners. The Laser Scanning Forum (https://www.laserscanningforum.com/) The mission of the Laser Scanning Forum is to become the world’s leading knowledge hub for 3D scanning and point cloud processing. Their applications are used for education, science, research, industry, commerce, and public service. This forum has many active users from around the world who are experts in various area of 3D scanning. It provides up-to-date news on hardware and software development as well as innovative ways to solve problems. In addition, this forum has a comprehensive list of software and hardware makers in the 3D scanning industry. The forum includes: - Peer-to-peer support in hardware and software - State-of-the-art new hardware and software features announcements - Tips on best practices - How-to guidance - Advanced research news Academic Research Academic research on applying ML on point cloud data processing is not the main focus of this PI. There is a significant amount of academic research on applying ML technologies for segmentation of point clouds. However, academic research does not often yield software readily available or useable for DOT users. Nevertheless, some ML point cloud segmentation software source codes are made available online. Below are a few articles that highlight state-of-the-art in applications of ML technologies in point cloud segmentation and feature extraction. Deep Learning for 3D Point Clouds: A Survey (IEEE TPAMI, 2020) (https://github.com/QingyongHu/SoTA-Point-Cloud) This survey presents a comprehensive review of recent deep learning methods for point cloud processing, including 3D point cloud analysis, 3D shape classification, 3D object detection, and 3D point cloud segmentation. Moreover, it presents results on several publicly available datasets. It is a very useful beginner’s guide on applying ML in point cloud data processing. PointCNN: Convolution on x-transformed points [6] (http://papers.neurips.cc/paper/7362-pointcnn-convolution-on-x-transformedpoints. pdf) Li et al. present a simple and general framework called PointCNN for feature learning from point clouds. CNN is the convolution operator leverage spatially-local correlation in data represented densely in grids (e.g. images). However, point clouds are irregular and unordered. Therefore, directly convolving kernels against features associated with the points will result in desertion of shape information and variance to point ordering. To address these problems, Li et al. proposed to learn an X-transformation from the input points to simultaneously promote two causes: the first is the weighting of the input features associated with the points, and the second is the permutation of the points into a latent and potentially canonical order. Element-wise product and sum operations of the typical convolution operator are subsequently applied on the X-transformed features. Their experiments show that PointCNN achieves on par or better performance than state-of-the-art methods. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (https://github.com/charlesq34/pointnet2) Charles R. Qi, Li (Eric) Yi, Hao Su, and Leonidas J. Guibas from Stanford University presented their research and Python code of PointNet++ deep learning on 3D point cloud classification and segmentation. 3D Semantic Segmentation of Large-Scale Point-Clouds in Urban Areas Using Deep Learning [7] (https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8687813) Mobile Laser Scanned Point-Clouds for Road Object Detection and Extraction: A Review [8] (https://www.mdpi.com/2072-4292/10/10/1531/pdf) 3D scenes semantic segmentation using deep learning based Survey [9] (https://iopscience.iop.org/article/10.1088/1757-899X/928/3/032083/pdf) This article reviewed different approaches for 3D semantic segmentation network that depend on deep learning neural network from 2017 to 2020. Toronto-3D: A Large-scale Mobile LiDAR Dataset for Semantic Segmentation of Urban Roadways [10] (https://openaccess.thecvf.com/content_CVPRW_2020/papers/w11/Tan_Toronto- 3D_A_Large- Scale_Mobile_LiDAR_Dataset_for_Semantic_Segmentation_of_CVPRW_2020_paper.pdf) Man vs. Computer: Benchmarking Machine Learning Algorithms for Traffic Sign Recognition [11] (https://www.ini.rub.de/upload/file/1470692859_c57fac98ca9d02ac701c/stallkampetal _gtsrb_nn_si2012.pdf) Contacts Caltrans contacted the individuals below to gather information for this investigation. We thank them for their time and input. State Agencies California Chris Thornton, PLS Chief, New Technology Systems Caltrans Chris.Thornton@dot.ca.gov Kellie Middlebrooks Associate Governmental Program Analyst (AGPA) Caltrans kellie.middlebrooks@dot.ca.gov Hawaii Goro Sulijoadikusumo Highways Planning Survey Engineer Hawaii Department of Transportation goro.sulijoadikusumo@hawaii.gov, 808-587-1839 Illinois Travis Lobmaster Systems Performance Manager Illinois Department of Transportation travis.lobmaster@illinois.gov Indiana Eric Banschbach Manager, Land & Aerial Survey Office Indiana Department of Transportation ebanschbach@indot.in.gov Maine Nick Dutil Transportation Engineer - LiDAR/UAS Maine Department of Transportation Nicholas.B.Dutil@maine.gov Michigan Frank Boston Remote Sensing Specialist Michigan Department of Transportation bostonf@michigan.gov, 517-331-7912 Minnesota Adam Smith Photogrammetric Unit Supervisor Minnesota Department of Transportation adam.smith@state.mn.us, 612-400-5298 Montana Chris Blumberg Photogrammetrist Montana Department of Transportation cblumberg@mt.gov, 406-461-2232 New Hampshire Deirdre Nash Assistant Research Engineer New Hampshire Department of Transportation deirdre.t.nash@dot.nh.gov, 603-271-8995 New Jersey Greg Krawtschenko Engineering Technician New Jersey Department of Transportation Greg.Krawtschenko@dot.nj.gov New York John A Becker Mapping Technologist New York State Department of Transportation john.becker@dot.ny.gov, 518-457-7759 Nevada Bill Nastasia Cartographer Nevada Department of Transportation vnastasia@dot.nv.gov North Carolina Brian Flippin Engineer III North Carolina Department of Transportation bflippin@ncdot.gov, 336-293-9633 North Dakota Derek Pfeifer Program Manager North Dakota Department of Transportation ddpfeifer@nd.gov Ohio Kyle Ince Transportation Engineer Ohio Department of Transportation Kyle.Ince@dot.ohio.gov, 614-466-2846 Oregon Christopher Glantz Deputy State Surveyor Oregon Department of Transportation christopher.glantz@odot.state.or.us, 503-986-3540 South Carolina Marvin Dawson Land Surveyor On-Call Contract Manager (SUE, UC, Aerial Mapping, Field Survey) South Carolina Department of Transportation dawsonmh@scdot.org West Virginia Travis Long Chief of Surveys West Virginia Department of Transportation travis.e.long@wv.gov Washington Marc Faucher Transportation Engineer 4 Washington State Department of Transportation fauchem@wsdot.wa.gov Survey Services Consultants Bradford Folta, Jr. GIS Architect & Chief Geomatics Officer Honey Badger Analytics LLC bradford.folta@honeybadgeranalytics.com, 218-330-0170 Jason Taylor Business Development Director – LiDAR, Seiler jason.taylor@seilerinst.com Dejan Vasic CEO DataDEV vasic@datadev-ds.com Steve Hutchinson Associate Director of Surveying Jacobs steve.hutchinson@jacobs.com Ben Vordermaier Australia and New Zealand Survey Manager Jacobs ben.vordermaier@jacobs.com Chris Sewell, Principal Surveyor Jacobs chris.sewell@jacobs.com, (+44)07909828980 References [1] Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen, “PointCNN: Convolution on xtransformed points,” Advances in neural information processing systems, vol. 31, pp. 820–830, 2018. [2] E. Che and M. Olsen, “An Efficient Framework for Mobile Lidar Trajectory Reconstruction and Mo-norvana Segmentation,” Remote Sensing, vol. 11, p. 836, Apr. 2019, doi: 10.3390/rs11070836. [3] E. Che and M. Olsen, “Fast Edge Detection and Segmentation Of Terrestrial Laser Scans Through Normal Variation Analysis,” ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. IV-2/W4, pp. 51–57, Sep. 2017, doi: 10.5194/isprs-annals-IV-2-W4-51-2017. [4] E. Che, M. J. Olsen, and J. Jung, “Efficient segment-based ground filtering and adaptive road detection from mobile light detection and ranging (LiDAR) data,” null, vol. 42, no. 10, pp. 3633–3659, May 2021, doi: 10.1080/01431161.2020.1871095. [5] E. Che, J. Jung, and M. Olsen, “Object Recognition, Segmentation, and Classification of Mobile Laser Scanning Point Clouds: A State of the Art Review,” Sensors, vol. 19, Feb. 2019, doi: 10.3390/s19040810. [6] J. Jung, E. Che, M. J. Olsen, and K. C. Shafer, “Automated and efficient powerline extraction from laser scanning data using a voxel-based subsampling with hierarchical approach,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 163, pp. 343–361, 2020. Appendix A: Caltrans Feature Extraction Requirements Line Features Table B-1 lists the line features that can be extracted. Features designated with a checkmark () are always extracted unless noted otherwise in the Survey Request. Table B-1: Line Features Feature Description Typically Extracted DTM Surface Type ADA detectable warning surface Ground Asphalt roadbed edges - bridge deck  Bridge deck Asphalt roadbed edges - ground  Ground Asphalt roadbed grade breaks - bridge deck  Bridge deck Asphalt roadbed grade breaks - ground  Ground Asphalt roadside edges  Ground Asphalt roadside grade breaks  Ground Barrier - bottom - all barrier types - bridge deck  Bridge deck Barrier - bottom - concrete - ground  Ground Barrier - bottom - K-rail - ground  Ground Barrier - face - thrie-beam, cable, other railing - ground - non-breakline  None Barrier - top - all barrier types - bridge deck Bridge deck Barrier - top - all barrier types - ground Ground Boundary outlining ground not visible areas (GNV) None Bridge abutment  Ground Bridge girder - bottom Bridge underside Bridge rail - bottom  Bridge deck Bridge soffit  Bridge underside Bridge wingwall  Ground Building faces including residential, commercial, bus stops, carports, decks, patios, etc. None Building overhangs including residential, commercial, bus stops, carports, decks, patios, etc. None Cabinet - outline  None Cattle guard  Ground Column, bent or pier - outline  None Concrete roadbed edges - bridge deck  Bridge deck Concrete roadbed edges - ground  Ground Concrete roadbed grade breaks - bridge deck  Bridge deck Feature Description Typically Extracted DTM Surface Type Concrete roadbed grade breaks - ground  Ground Concrete roadside edges  Ground Concrete roadside grade breaks  Ground Conduit None Crash cushion - multiple in a line or outline None Culvert - flowline  Ground Culvert - top and bottom  Ground Culvert wingwall  Ground Curb in roadbed areas - flowline - bridge deck  Bridge deck Curb in roadbed areas - flowline - ground  Ground Curb in roadbed areas - lip - ground  Ground Curb in roadbed areas - top - bridge deck  Bridge deck Curb in roadbed areas - top - ground  Ground Curb, miscellaneous - flowline  Ground Curb, miscellaneous - lip  Ground Curb, miscellaneous - top  Ground Dike in roadbed areas - bottom  Ground Dike in roadbed areas - top  Ground Dike, miscellaneous - bottom  Ground Dike, miscellaneous - top  Ground Dirt roadbed edges  Ground Dirt roadbed grade breaks  Ground Dirt roadside edges  Ground Dirt roadside grade breaks  Ground Drainage inlet - outline - bridge deck  Bridge deck Drainage inlet - outline - ground  Ground Fence  None Flowlines - roadbed - except curb flowlines and valley gutters - bridge deck  Bridge deck Flowlines - roadbed - except curb flowlines and valley gutters - ground  Ground Flowlines - roadside  Ground Flowlines from unregistered point cloud data in study ground DTM areas Flowlines of all water and drainage facilities except culverts  Ground Gate  None Guy wire - line from anchor to post  None Headwall - top and bottom  Ground Linework indicating the direction to or from a feature None Mailboxes - multiple in a line  None Feature Description Typically Extracted DTM Surface Type Marker - pavement - reflective and non-reflective - multiple in a line None Marker - utility None Miscellaneous bridge deck linear features Bridge deck Miscellaneous bridge underside linear features Bridge underside Miscellaneous hydrographic linear features including slope protection, etc.  Ground Miscellaneous roadbed edges - except asphalt, concrete, and dirt - bridge deck  Bridge deck Miscellaneous roadbed edges - except asphalt, concrete, and dirt - ground  Ground Miscellaneous roadbed grade breaks - except asphalt, concrete, and dirt - bridge deck  Bridge deck Miscellaneous roadbed grade breaks - except asphalt, concrete, and dirt - ground  Ground Miscellaneous roadside edges - except asphalt, concrete, dirt, and rock  Ground Miscellaneous roadside grade breaks - except asphalt, concrete, dirt, and rock  Ground Miscellaneous structures linear features not on the bridge including slope protection, etc. None Miscellaneous utilities linear features None Miscellaneous vegetation linear features None Open drainage facilities including banks, canals, catch basins, ditches, spillways - all aspects except flowlines  Ground Orchard - outline  None Overhead facility None Overside drain  Ground Pull box - outline  None Pump - pump and pump house outline  None Railroad ballast  Ground Railroad rail - individual rail for vertical and horizontal clearance requirements  Ground Railroad track - displays both rails where the right rail is the true location and the left rail is an approximate location  None Random breaklines in bridge deck DTM areas Bridge deck Random breaklines from unregistered point cloud data in study bridge deck DTM areas Study Random breaklines in bridge underside DTM areas Bridge underside Random breaklines from unregistered point cloud data in study bridge underside DTM areas Study Random breaklines in ground DTM areas Ground Random breaklines from unregistered point cloud data in study ground DTM areas Study Rock roadbed edges  Ground Feature Description Typically Extracted DTM Surface Type Rock roadbed grade breaks  Ground Rock roadside edges  Ground Rock roadside grade breaks  Ground Sidewalk - ground  Ground Sign - cantilever - line from post to end of sign  None Sign - multi-post - line from post to post  None Sign - overhead bridge - line from post to post  None Slide - edges  Ground Striping - lane line stripes - bridge deck  Bridge deck Striping - lane line stripes - ground  Ground Striping - dashed & solid except for fog stripes - bridge deck  Bridge deck Striping - dashed & solid except for fog stripes - ground  Ground Striping - fog stripes along ETW - bridge deck  Bridge deck Striping - fog stripes along ETW - ground  Ground Tank - outline  None Transmission tower - outline  None Underground utility facility - markouts and actual location  None Vault - outline  None Vineyard - rows and outline  None Wall - bottom - masonry including sound walls and other non-retaining walls  Ground Wall - bottom - retaining  Ground Wall - top - masonry and retaining None Water body edges - open water features including lakes, ponds, pools, rivers, streams - edges, high water marks, and threads  Ground Water body edges from unregistered point cloud data in study ground DTM areas Ground Weir None Point Features Table B-2 lists the point features that can be extracted. Features designated with a checkmark () are always extracted unless noted otherwise in the Survey Request. Features designated with a checkmark in a box () should only be extracted when the line feature is not required. Table B-2: Point Features Feature Description Typically Extracted DTM Surface Type Barrier - guardrail terminator - left None Barrier - guardrail terminator - right None Bollards, protective pipes, etc.  None Cabinet - center  None Callbox  None Cleanout - drain, sewer, storm, pool, etc.  None Closed circuit TV camera None Crash cushion None Culvert - flowline single location  Ground Culvert - top and bottom single location  None Drainage inlet - center - rectangular - bridge deck  None Drainage inlet - center - rectangular - ground  None Drainage inlet - center - round - bridge deck  None Drainage inlet - center - round - ground  None Electrolier  None Fire hydrant  None Flag pole  None Flared end section - lip end None Gate post  None Guy anchor  None High water mark None Hose bib None Lighting - decorative lamp post, electrolier, landscape lighting, light fixtures, recessed lights, etc.  None Loop detector None Mailbox - single location  None Manhole - center  None Marker - guidepost None Marker - miscellaneous None Marker - pavement None Marker - utility  None Meter - center  None Miscellaneous bridge deck point features None Miscellaneous bridge underside point features None Miscellaneous hydrographic point features None Miscellaneous roadbed point features Ground Miscellaneous roadside point features None Feature Description Typically Extracted DTM Surface Type Miscellaneous structures point features None Miscellaneous utilities point features None Miscellaneous vegetation point features None Motor vehicle pullout None Parking meter None Pedestal - center  None Pedestrian button None Pedestrian button pole  None Pedestrian signal  None Pole - center  None Pullbox - center - rectangular  None Pullbox - center - round  None Pump - center  None Railroad Appurtenances including control box, switch, etc.  None Railroad gate post  None Railroad signal  None Sign - cantilever - post location  None Sign - multi-post - center of sign  None Sign - post location - overhead bridge  None Sign - post location - single post  None Slide - monitoring point None Spot elevations & mass points in bridge deck DTM areas Bridge deck Spot elevations & mass points from unregistered point cloud data in study bridge deck DTM areas Study Spot elevations & mass points in bridge underside DTM areas Bridge underside Spot elevations & mass points from unregistered point cloud data in study bridge underside DTM areas Study Spot elevations & mass points in ground DTM areas Ground Spot elevations & mass points from unregistered point cloud data in study ground DTM areas Study Sprinkler None Standpipe  None Tank - center  None Telephone  None Feature Description Typically Extracted DTM Surface Type Traffic signal - pole location  None Transmission tower - center  None Tree  None Underground utility facility - markouts and feature - single location  None Utility appurtenances  None Valve  None Vault - center  None Vent - center - unknown or other None Well None Appendix B: Automated LiDAR Point Cloud Feature Extraction Software Questionnaire