Revolutionizing Patient Care: How a Text-Based Chatbot is Transforming the Hospital Experience

In today’s fast-paced healthcare environment, patients often struggle to get the information and support they need during their hospital stay. Overworked medical staff may not always be available to answer questions or provide guidance, leaving patients feeling anxious and unsupported.

This is where a text-based healthcare chatbot can make a significant difference. By providing 24/7 access to relevant information and personalized support, chatbots can help alleviate patient concerns, reduce the burden on healthcare workers, and improve overall patient satisfaction and outcomes.

Why is there a need for Chatbots in Healthcare?

Global health crises lay bare the weaknesses in medical systems around the world. Just a few years ago, the Covid-19 pandemic exposed the vulnerabilities in healthcare systems worldwide. A study by the World Health Organization found that 90% of countries experienced disruptions to essential health services due to the pandemic.

In the United States alone, hospitals saw a 21% increase in patient volume during Covid surges, while simultaneously facing critical staffing shortages as healthcare workers fell ill. A survey of over 1,300 healthcare professionals revealed that 93% experienced stress, 86% reported anxiety, and 76% reported exhaustion and burnout.

An unprecedented strain on medical resources can leave many patients without adequate care and support. Chatbots have emerged as a scalable solution to bridge this gap, capable of providing 24/7 access to health information and guidance. 

Our Goal

This project focused on solving this issue by creating solutions in multiple forefronts as below:

1. Creating a chatbot to help users get answers at their fingertips
2. Creating a dashboard for users to visualize insights on a particular disease
3. Performing sentiment analysis on the data gathered from responses from patients to visualize the emotions into positive, negative, and neutral categories and improve the chatbot experience based on the results.

So How Do We Proceed?

Project Pipeline

Data Collection and Preprocessing

We started by web scraping reliable medical pages to collect data on 6 diseases for the chatbot’s knowledge base. We had individual .csv files with 21 columns of data for each disease, covering topics like symptoms, treatment, facts, expenses, and more.

Covid	Symptoms (male, female, babies, common, old-people, critical-stage)
Diabetes	Treatment ( general, babies, old people, critical-stage)
Respiratory Diseases	Medication (general, babies, old people)
Heart Diseases	Facts or Myths
Dengue	Expenses
Diarrhoeal Diseases	Funding
	Survival Rate
	Diagnosis
	FAQ
	Prevention

After merging the CSV files, we preprocessed the data by:

For patient experience, we used the Twitter API to gather 63,239 tweets without class and 29,706 tweets with class. The collected tweets needed cleaning, such as:

The image below shows the conditions for cleaning the tweets:

The resulting dataframe contains the original tweets, cleaned tweets, and their classes.

NLP Sentiment Analysis

Sentiment Analysis determines the emotional tone behind words to understand attitudes, opinions, and emotions in online mentions.

Our project classified tweets based on patients’ experiences during treatment, such as their feelings and the helpfulness of hospital staff.

VADER (Valence Aware Dictionary and Sentiment Reasoner) was an excellent choice because our data was unlabeled for sentiments, and it handles preprocessing, acronyms, and emoticons. VADER is a lexicon and rule-based sentiment analysis tool attuned to social media sentiments. It provides the sentiment score and the degree of positivity, negativity, or neutrality.

Step 1: Active Learning For Labeling The Data As “Relevant” Or “Irrelevant”

Step 2: Filtering The Data To Make The Corpus Customized To The Project Idea

Number of Relevant and Irrelevant Tweets

Step 3: Performing Sentiment Analysis For 3 Sentiments Using Vader

Step 4: Visualization using Python

Word Cloud is a data visualization technique used for representing text data in which the size of each word indicates its frequency or importance. In simpler terms, a ‘word cloud’ is a visual representation of word frequency.

Word clouds are increasingly being employed as a simple tool to identify the focus of written material. They are widely used for analyzing data from social network websites.

Data Visualization and Dashboarding

Data visualization, which was once a relatively obscure field, has now become ubiquitous in the fields of business intelligence and data journalism. There are numerous methods for effectively and beautifully presenting insights, as well as a plethora of blogs dedicated to creating and analyzing visualizations. Dashboards are what we utilized to represent data in a variety of ways.

Storytelling with Data

Data visualization exists, in large part, to help create a compelling narrative. In the Health Dashboard, we are trying to show the level of data based on the Disease Conditions. Simultaneously, data talks about Sentiment Analysis that is extracted from tweets data.

Innovative Data Visualization

Rather than limiting data to mundane functionality, we can take advantage of the opportunity to showcase innovative data visualization projects. To drill down into the data, the health dashboard makes use of GIF images, Tabular, Filters, Actions, and Word Cloud. In Addition, the Sentiment Analysis Dashboard uses Word Cloud, Bar Charts, Dual Axis Charts, and Filter Actions.

Dashboards

Users can visualize the symptoms, diagnosis, treatment, expenses, and FAQ’s in the Health Conditions Dashboard based on the disease they had chosen. It was developed to help patients and their families understand and learn more about the diseases and data used by our chatbot. Furthermore, the Sentiment Analysis Dashboard is intended to help doctors and medical administrators gain a better understanding of the experiences of patients and the general public when they visit hospitals.

Chatbot Modeling Rasa Framework

In a Chatbot implementation, we can integrate to provide advanced conversational capabilities for healthcare professionals. The mechanism involves training the Chatbot by using Generative AI technology, which enables it to understand and respond to natural language queries related to healthcare. By incorporating advanced NLP techniques, the ChatBot can unravel the intricacies of complex medical documents, unlocking hidden knowledge and providing valuable insights to users in a conversational manner.

We chose RASA to build our chatbot for its versatility, functionality, and scalability for future healthcare applications. As an open-source AI conversational chatbot tool, Rasa is regularly updated by a huge community of developers and users.

Now, how does Rasa work?

If you didn’t quite grasp the image above, no worries! We’ll break it down in a simple way. The Rasa structure is essentially divided into three main files: intents.yml, stories.yml, and domain.yml.

Intents are hard-coded examples of user inputs that train Rasa to anticipate user messages. A simple intent could be “Hi.” Entities are extracted from the user’s input and stored in slots for later use. For example, if a user mentions “Covid,” it’s captured as a ‘disease’ entity and stored for fetching data or incorporating into the conversation.

Stories contain generalized conversational flows between the user and chatbot. They can be triggered based on the user’s input to respond to their query. In this project, we have stories for Symptoms, Cure, Medication, Expenses, Greet, and Goodbye to make the user feel at home.

Lastly, domain.yml oversees the entire process, storing data related to entities, slots, stories, user responses, and actions.

Deployment

Up until this point, we were developing our chatbot locally. But to progress to the next stage, we needed to deploy our assistant and make it accessible to the world. The Rasa stack can run on-premise or on your own cloud account, giving us flexibility in deploying our assistant.

These were the deployment methods we explored:

Using ngrok

ngrok is a cross-platform application that exposes local server ports to the internet. It offers various advantages as it is easy to use, fast, supports UTP and HTTPS, and also it can be used for free.

Using Heroku and Docker Compose

Heroku is a container-based cloud Platform as a Service(PaaS). Its popularity has always hinged on its simplicity, elegance, and usability. Docker Compose is a tool for running multi-container applications on a single host machine. Docker Compose networks the containers together, allowing the services to reach each other and making it possible to start or stop all containers with a single command.

Potential Benefits

When the three major outcomes of these projects are implemented at full scale, we could see the following benefits:

Large Knowledge Models: By applying Large Knowledge Models (LKMs), we can derive solutions from extensive language models to comprehensively understand medical terminologies, nuanced phrasing, and intricate medical documentation. We can customize this technology to meet the specific needs of our clients by tailoring the language models and training data to their unique healthcare scenarios.

Cost Estimation: The system can be configured to provide patients with cost-estimation in payor market and health insurance by asking questions in natural language, along with expansions of the NLP based chatbot into other healthcare processes. 

Instant Access to Health Information: The chatbot will help users get medical and health-related information at their fingertips, without needing to wait for doctors or health experts to answer general questions.

Enhanced Data Visualization and Transparency: The dashboard will enable users to visualize health and disease-related data utilized by the chatbot in real-time. It will also help answer questions related to the chatbot’s reasoning and provide better feedback on patient and user behavior to doctors and hospital administration.

Improved Emotional Support: The sentiment analysis provides insights into how users’ emotions are classified as positive, negative, or neutral. This will help determine the level of stress, anxiety, and nervousness among patients and users, allowing the chatbot and dashboard to work together to reduce these negative emotions.

Future Scope

Training on More Data: Improving the Rasa chatbot by training it on more data and incorporating advanced Natural Language Understanding (NLU) techniques to make it more robust and accurate in understanding and responding to user queries.

Integrating Sentiment Analysis: Incorporate sentiment analysis capabilities into the chatbot to enable it to respond to patient inquiries based on the sentiment detected in the preceding remark or query, allowing for more empathetic and personalized interactions.

Expanding Disease Database: Adding more comprehensive databases of diseases and their related attributes to expand the dataset used by the chatbot, making it more generalizable and applicable in a wider range of clinical settings.

Implementing Age-Based Filters: Improving the dashboard by implementing age-based filters using actions that dynamically change the attributes of the health conditions according to the selected age filter, showcasing relevant features while hiding others based on the user’s age group.

Potential Applications in Other Industries

While this project focused on developing a chatbot, dashboard, and sentiment analysis system for the healthcare industry, the underlying technologies and approaches have broad applicability across various sectors. Here are some potential applications in other industries:

Customer Service: Chatbots powered by natural language processing (NLP) can revolutionize customer service by providing instant, 24/7 support to customers. These chatbots can handle common queries, troubleshoot issues, and escalate complex problems to human agents when necessary. By leveraging sentiment analysis, businesses can also gauge customer satisfaction in real-time and proactively address concerns.

Education: Chatbots can serve as virtual tutors or learning companions, assisting students with their coursework, answering questions, and providing personalized feedback. Educational institutions can also use dashboards to visualize student performance data, identify areas for improvement, and make data-driven decisions to enhance the learning experience.

Finance: In the financial sector, chatbots can help customers with account inquiries, transaction history, and even provide personalized investment advice based on user preferences and risk tolerance. Sentiment analysis can be applied to news articles, social media posts, and market data to gauge market sentiment and inform investment strategies.

E-commerce: Online retailers can deploy chatbots to assist customers with product recommendations, order tracking, and returns or exchanges. By analyzing customer sentiment through reviews and feedback, e-commerce businesses can identify trends, improve product offerings, and optimize their marketing efforts.

Human Resources: HR departments can leverage chatbots to streamline employee onboarding, answer common HR-related questions, and provide guidance on company policies and benefits. Sentiment analysis can be used to monitor employee sentiment, identify potential issues affecting morale, and inform initiatives to improve employee engagement and retention.