SB Technology conducts demonstration experiment of generative AI using RAG on JR West's construction department data

SB Technology Corp. (Head office: Shinjuku-ku, Tokyo; President ＆ CEO: Shinichi Ata; hereinafter referred to as SBT) has conducted a search expansion generation (RAG) analysis of construction department data of West Japan Railway Company (Head office: Osaka City, Osaka Prefecture; President and CEO: Hasegawa Kazuaki; hereinafter referred to as JR West).※1 We conducted a demonstration experiment of generative AI using this technology.
The purpose of this demonstration experiment is to verify the usefulness of the accuracy of answers generated by generative AI for the vast amount of documents related to railway construction work held by JR West's construction department. By utilizing data that contains accumulated past knowledge and technical know-how, we aim to improve the accuracy of technical judgments of young engineers.
Utilizing SBT 's know-how in generative AI, a verification system was developed in one month that imported approximately 60,000 files representing part of the construction sector data. In verifying the accuracy of responses provided by RAG, the system was found to be useful in classifying information and generating responses based on the classification.

■Background

In the "JR West Group Digital Strategy," JR West is promoting the digitalization and utilization of vast amounts of data, with the goal of rebuilding customer experience, railway systems, and employee experience. In addition to the aspect of knowledge transfer between staff, the Osaka Construction Office, which is responsible for JR West's railway construction projects, is promoting efforts to pass on skills by managing and sharing construction data related to railways and technical know-how on the cloud storage service Box, in anticipation of the organization's younger generation becoming more involved in the future, as experienced engineers have been reaching retirement age in recent years. However, the time and effort required to search through the vast amount of material for information on past construction cases and safety and quality rules was a major burden, and the situation was such that the company still had to rely on the experience and knowledge of experienced engineers. This time, as part of the "rebuilding of the employee experience," technical capabilities that could solve these business issues were required.
SBT has an extensive track record of implementing Microsoft products, and has extensive know-how regarding generative AI, including its own use of generative AI and the provision of the corporate generative AI service "dailyAI." Utilizing this knowledge, SBT built an information search system using RAG and conducted a demonstration experiment to confirm the usefulness of the accuracy of answers.

■ Details of the demonstration experiment (December 2023 to March 2024)

〇 Building a verification system environment for generative AI
We built a generative AI verification system using Azure OpenAI Service on the Microsoft Azure cloud infrastructure in about one month. In addition to security protection configured with private access via Azure Virtual Network, we implemented security measures to automatically delete personal information before linking to the generative AI, and built it as a secure platform for using generative AI for approximately 60,000 files stored in Box storage.
The verification system allows users to enter prompts (instructions) to generate answers based on relevant information from construction sector data, and various parameters can be used to adjust the answer tendencies of the generating AI.

RAG function development and accuracy verification
The data to be verified this time included various types of documents related to railway construction work, such as work regulations, past accident information, and construction plans, and there was variation in the way the contents were written depending on who created them (e.g., rail vs. track, concrete vs. RC). Therefore, in order to absorb and take into account these variations in writing and further improve the accuracy of information searches, verification was carried out using RAG. Also, to improve convenience when searching, in addition to presenting related document links that serve as citations for generated answers, a function to automatically generate document tags that succinctly understand the contents of a file (e.g., #machinery equipment, #safety assurance). In developing these, agile development was used to incorporate the latest technology in generative AI, which is constantly evolving.※2 Using this method, we were able to add and improve functions in short cycles.

Implemented RAG (partial excerpt)

Please scroll left and right to view.

■ Verification results

We comprehensively verified the items that served as benchmarks for the verification, including convenience, such as appropriate classification of various types of documents such as construction plans, business reference materials, and contract documents, and the answer generation content according to the classification, and confirmed a certain level of usefulness.

■Comment from Hiroyuki Masaki, General Manager of Railway DX, Innovation Headquarters, West Japan Railway Company

JR West has formulated the JR West Group Digital Strategy and is working on restructuring the three areas of (1) customer experience, (2) railroad systems, and (3) employee experience. In (3), we would like to thank SB Technology Corp. for their cooperation in conducting a demonstration experiment on a system that quickly collects the information we want from the vast amount of past technical information and construction know-how in railroad construction work, and combines, summarizes, and expresses this information.
In the future, we will work to further deepen the cooperation between the two companies, aiming to make this a use case that will serve as a reference for people involved in construction projects around the world, beyond the railroad industry.

■About the future

In addition to the results of this PoC, we will continue to support JR West in resolving various operational issues by leveraging our knowledge of Microsoft products, generative AI, and SBT 's know-how regarding security.

■ Related
Azure OpenAI Service Management Infrastructure Construction Service
https://www.softbanktech.co.jp/service/list/microsoft-azure/azure-openai-management/

• *1 RAG (Retrieval-Augmented Generation): A technology that searches for highly relevant information and generates answers based on that information.
• *2 A software development methodology that quickly reflects feedback and executes a cycle of planning, development, testing, and improvement in a short period of time.

• *The company names, product names, and service names mentioned in this release are trademarks or registered trademarks of our company or the respective companies or organizations.
• * Microsoft and Azure are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
• *Azure OpenAI Service is the name of a service provided by Microsoft Corporation.