It is the year 2020 and students are experiencing a new way of life when it comes to getting an education. Students are realizing they need to adopt a proactive and self-service mindset in to fulfill their academic needs. An intelligent chatbot that helps students find and access learning content supports this new self-service model. This pattern shows users how to build a self-service chatbot not only for education, but also for any other industries where users need to find information quickly and easily.
Using Watson Assistant, this pattern defines a dialog that a student and a course provider might experience as a student searches for learning content. Students can input grade-level and academic topics question, and the chatbot responds with course recommendations and learning content links. The conversation responses are further enhanced by using Watson Discovery and the Watson Assistant Search skill. Natural Language Understanding (NLU) is introduced in this pattern to complement Watson Discovery's accuracy by extracting custom fields for entities, concepts, and categories.
An Assistant search skill is a mechanism that allows you to directly query a Watson Discovery collection from your Assistant dialog. A search skill is triggered when the dialog reaches a node that has a search skill enabled. The user query is then passed to the Watson Discovery collection via the search skill, and the results are returned to the dialog for display to the user. Customizing how your documents are indexed into Discovery improves the answers returned from queries and what your users experience.
Click here for more information about the Watson Assistant search skill.
NLU performs text analysis to extract metadata such as concepts, entities, keywords, and other types of categories of words. Data sets are then enriched with NLU-detected entities, keywords, and concepts (for example, course names). Although Discovery provides great results, sometimes a developer finds that the results are not as relevant as they might be and that there is room for improvement. Discovery is built for "long-tail" use cases where the use case has many varied questions and results that you can't easily anticipate or optimize. Additionally, if the corpus of documents is relatively small (less than 1000), Discovery doesn't have enough information to distinguish important terms and unimportant terms. Discovery can work with a corpus this small - but it is less effective because it has less information about the relative frequency of terms in the domain.
.csv
file.csv
to .json
files (required for the Discovery collection)Follow these steps to setup and run this code pattern. The steps are described in detail below.
bash
git clone https://github.com/IBM/Education-SelfService-AI-Assistant
Create the following services:
NOTE: use the
Plus
offering of Watson Assistant. You have access to a 30 day trial.
env-sample
file and call it .env
Add in your NLU and Discovery credentials. NLU enriches Discovery by creating the addition of metadata tags to your data sets. In other words, includes terms that overlap with words that users might actually provide in their queries.
.csv
files through NLU and extract entities and concepts. Do this by running the python program:bash
cd src
pip install watson-developer-cloud==1.5
pip install --upgrade ibm-watson
pip install pandas
sudo pip3 install -U python-dotenv
python NLUEntityExtraction.py
Note that this may take a few minutes. This will create 2
.csv
files. Take a look at the format by exploring the files.
Watson Discovery uses AI search technology to retrieve answers to questions. It contains language processessing capabilities and can be trained on both structured and unstructured data. The data that Discovery is trained on is contained within what is called a Collection
(aka Database). You can learn more about Watson Discovery here
First create a Discovery Collection. This is the database that holds your response data.
Find the Discovery service in your IBM Cloud Dashboard.
Launch Watson Discovery
.Upload your own data
button. You will see that you have one collection created that comes with Discovery by default. That is the Watson Discovery News
collection. Courses
English
languageCreate
You will need to configure Discovery in 2 steps:
Create a set of .json
files that Discovery will consume for its collection. The node program converts the .csv
file to a set of .json
files in a directory named manualdocs
.
Install Node.js (Versions >= 6).
In the root directory of your repository, install the dependencies.
bash
npm install
Run below command
bash
node read-file.js
Verify the JSON
files exists.
Programmatically upload the .json
files into the discovery collection
ensure you have added your Discovery credentials into a .env file sitting in your root directory of your repo.
bash
npm install python-dotenv
npm install ibm-watson
node upload-file.js
This will take approximately 5 minutes or less. Documents are uploaded to your Courses Collection. Take a look and explore the collection. You should see about 135 documents.
NOTE: If using the Discovery Lite plan, you are limited to loading up to 1000 files into your discovery service. This limit is not per collection, but the combined number for all collections in your service.
Find the Assistant service in your IBM Cloud Dashboard.
Click on the service and then click on Launch Watson Assistant
.
Go to your Assistant tab and click create assistant
.
You will see that you now have the ability to add a: * Dialog skill * Search skill
Add dialog skill
Note: You will import the
Dialog skill
that is saved within this repo:data/assistant/Student-skill-dialog.json
.
Click on Import Skill
Click on Choose JSON File
. Go to your cloned repo dir, and Open
the JSON file in data/assistant/Student-skill-dialog.json
Click on Import
You will see that the dialog has been imported. Click on the words Student-skill
and explore the dialog to see the intents, entities, dialog. You can also add to the dialog to extend the conversation.
Note: you can also create your dialog from scratch by selecting
Create skill
and adding in your own intents, entities and dialog.
From your Assistant panel, click on Add search skill
.
Note: If you have provisioned Watson Assistant on IBM Cloud, the search skill is only offered on a paid plan, but a 30-day trial version is available if you click on the
Plus
button.
From the Add Search Skill
panel, select the Create skill
tab.
Give your search skill a unique name, then click Continue
.
From the search skill panel, select the Discovery service instance and collection you created previously.
Note Assistant can only connect to 1 Discovery collection at a time. You will use either the elementary or high school collections and then discovery will respond appropriately.
Click Configure
to continue.
From the Configure Search Response
panel, select name
as the field to use for the Title
of the response. Select Description
as the field for the Body
and select URL
as the filled to use foor URL
. Click Save
to complete the configuration.
Now when the dialog skill node invokes the search skill, the search skill will query the Discovery collection and display the text result to the user.
Learn more about the assistant search skill here
Settings
.
Search Skill
tab and ensure that is in Enabled
.NOTE: The following feature is currently only available for Watson Assistant provisioned on IBM Cloud.
Normally, you can test the dialog skill be selecting the Try it
button located at the top right side of the dialog skill panel, but when integrated with a search skill, a different method of testing must be used.
From your assistant panel, select Add Integrations
.
From the list of available integration types, select Preview link
.
From the Preview link integration
panel, name your preview link and click Create
.
If you click on the generated URL link, you will be able to interact with your dialog skill. Note that the input "Can you recommend a math class?" has triggered our Which grade do you want recommendation for
dialog node and invoked our search skill.
Error: If you get an error that says you
> Traceback (most recent call last):
> File "NLUEntityExtraction.py", line 4, in
you need to make sure you are using the same verion of pip and python. We recommend you use Python 3 and Pip 3
Error: Unable to list workspaces for Watson Assistant: Forbidden: Access is denied due to invalid credentials.
This error occurs with
Deploy to IBM Cloud
button. Configure a runtime environment variable forASSISTANT_APIKEY
to allow automatic configuration of the default skill or configureSKILL_ID
to use another skill.
To work with a free trial, a small free Discovery environment is created. If you already have a Discovery environment, this will fail. If you are not using Discovery, check for an old service thay you may want to delete. Otherwise use the .env DISCOVERY_ENVIRONMENT_ID to tell the app which environment you want it to use. A collection will be created in this environment using the default configuration.
This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache Software License, Version 2.
Apache Software License (ASL) FAQ
Bumps json5 from 2.1.3 to 2.2.3.
Sourced from json5's releases.
v2.2.3
- Fix: [email protected] is now the 'latest' release according to npm instead of v1.0.2. (#299)
v2.2.2
- Fix: Properties with the name
__proto__
are added to objects and arrays. (#199) This also fixes a prototype pollution vulnerability reported by Jonathan Gregson! (#295).v2.2.1
- Fix: Removed dependence on minimist to patch CVE-2021-44906. (#266)
v2.2.0
Sourced from json5's changelog.
v2.2.3 [code, diff]
- Fix: [email protected] is now the 'latest' release according to npm instead of v1.0.2. (#299)
v2.2.2 [code, diff]
- Fix: Properties with the name
__proto__
are added to objects and arrays. (#199) This also fixes a prototype pollution vulnerability reported by Jonathan Gregson! (#295).v2.2.1 [code, diff]
- Fix: Removed dependence on minimist to patch CVE-2021-44906. (#266)
v2.2.0 [code, diff]
c3a7524
2.2.394fd06d
docs: update CHANGELOG for v2.2.33b8cebf
docs(security): use GitHub security advisoriesf0fd9e1
docs: publish a security policy6a91a05
docs(template): bug -> bug report14f8cb1
2.2.210cc7ca
docs: update CHANGELOG for v2.2.27774c10
fix: add proto to objects and arraysedde30a
Readme: slight tweak to intro97286f8
Improve example in readmeDependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Bumps qs from 6.5.2 to 6.5.3.
Sourced from qs's changelog.
6.5.3
- [Fix]
parse
: ignore__proto__
keys (#428)- [Fix]
utils.merge
: avoid a crash with a null target and a truthy non-array source- [Fix] correctly parse nested arrays
- [Fix]
stringify
: fix a crash withstrictNullHandling
and a customfilter
/serializeDate
(#279)- [Fix]
utils
:merge
: fix crash whensource
is a truthy primitive & no options are provided- [Fix] when
parseArrays
is false, properly handle keys ending in[]
- [Fix] fix for an impossible situation: when the formatter is called with a non-string value
- [Fix]
utils.merge
: avoid a crash with a null target and an array source- [Refactor]
utils
: reduce observable [[Get]]s- [Refactor] use cached
Array.isArray
- [Refactor]
stringify
: Avoid arr = arr.concat(...), push to the existing instance (#269)- [Refactor]
parse
: only need to reassign the var once- [Robustness]
stringify
: avoid relying on a globalundefined
(#427)- [readme] remove travis badge; add github actions/codecov badges; update URLs
- [Docs] Clean up license text so it’s properly detected as BSD-3-Clause
- [Docs] Clarify the need for "arrayLimit" option
- [meta] fix README.md (#399)
- [meta] add FUNDING.yml
- [actions] backport actions from main
- [Tests] always use
String(x)
overx.toString()
- [Tests] remove nonexistent tape option
- [Dev Deps] backport from main
298bfa5
v6.5.3ed0f5dc
[Fix] parse
: ignore __proto__
keys (#428)691e739
[Robustness] stringify
: avoid relying on a global undefined
(#427)1072d57
[readme] remove travis badge; add github actions/codecov badges; update URLs12ac1c4
[meta] fix README.md (#399)0338716
[actions] backport actions from main5639c20
Clean up license text so it’s properly detected as BSD-3-Clause51b8a0b
add FUNDING.yml45f6759
[Fix] fix for an impossible situation: when the formatter is called with a no...f814a7f
[Dev Deps] backport from mainDependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Bumps codecov from 3.6.5 to 3.8.3.
Sourced from codecov's releases.
v3.8.3
Fixes
- #329 fix: Test if response has two lines
Dependencies
- #306 Bump eslint-config-prettier from 7.2.0 to 8.3.0
- #305 Bump eslint from 7.21.0 to 7.25.0
- #302 Bump mock-fs from 4.13.0 to 4.14.0
- #308 Bump lodash from 4.17.19 to 4.17.21
- #309 Bump ignore-walk from 3.0.3 to 3.0.4
- #310 Bump hosted-git-info from 2.8.8 to 2.8.9
- #325 Bump prettier from 2.2.1 to 2.3.2
- #326 Bump actions/setup-node from 2.1.5 to 2.2.0
- #328 Bump lint-staged from 10.5.4 to 11.0.1
- #330 Bump eslint from 7.25.0 to 7.31.0
- #331 Bump ws from 7.3.1 to 7.5.3
- #332 Bump urlgrey from 0.4.4 to 1.0.0
- #334 Bump husky from 6.0.0 to 7.0.1
- #333 Bump teeny-request from 7.0.1 to 7.1.1
v3.8.2
3.8.2
Fixes
- #304 Add coverage-final.json as a possible coverage file during file lookup
v3.8.1
Fixes
- #246 Revert "Bump teeny-request from 6.0.1 to 7.0.0"
v3.8.0
Features
- #160 Add Github Actions support
Fixes
- #173 Fix broken gcov command
- #195 Update Node testing versions
- #200 Remove flaky tests
- #204 Create CHANGELOG and remove flaky v4 test
- #208 Add license scan report and status
- #220 Remove errant bitly
Dependencies
... (truncated)
Sourced from codecov's changelog.
3.8.3
Fixes
- #329 fix: Test if response has two lines
Dependencies
- #306 Bump eslint-config-prettier from 7.2.0 to 8.3.0
- #305 Bump eslint from 7.21.0 to 7.25.0
- #302 Bump mock-fs from 4.13.0 to 4.14.0
- #308 Bump lodash from 4.17.19 to 4.17.21
- #309 Bump ignore-walk from 3.0.3 to 3.0.4
- #310 Bump hosted-git-info from 2.8.8 to 2.8.9
- #325 Bump prettier from 2.2.1 to 2.3.2
- #326 Bump actions/setup-node from 2.1.5 to 2.2.0
- #328 Bump lint-staged from 10.5.4 to 11.0.1
- #330 Bump eslint from 7.25.0 to 7.31.0
- #331 Bump ws from 7.3.1 to 7.5.3
- #332 Bump urlgrey from 0.4.4 to 1.0.0
- #334 Bump husky from 6.0.0 to 7.0.1
- #333 Bump teeny-request from 7.0.1 to 7.1.1
3.8.2
Fixes
- #304 Add coverage-final.json as a possible coverage file during file lookup
3.8.1
Fixes
- #246 Revert "Bump teeny-request from 6.0.1 to 7.0.0"
3.8.0
Features
- #160 Add Github Actions support
Fixes
- #173 Fix broken gcov command
- #195 Update Node testing versions
- #200 Remove flaky tests
- #204 Create CHANGELOG and remove flaky v4 test
- #208 Add license scan report and status
- #220 Remove errant bitly
Dependencies
... (truncated)
e22061b
Merge pull request #335 from codecov/3.8.3981df8b
3.8.3135555c
Merge pull request #333 from codecov/dependabot/npm_and_yarn/teeny-request-7.1.165b53a3
Merge pull request #334 from codecov/dependabot/npm_and_yarn/husky-7.0.16e4af4d
Bump teeny-request from 7.0.1 to 7.1.11149168
Merge pull request #332 from codecov/dependabot/npm_and_yarn/urlgrey-1.0.0883785c
Merge pull request #331 from codecov/dependabot/npm_and_yarn/ws-7.5.304d5ff7
Merge pull request #330 from codecov/dependabot/npm_and_yarn/eslint-7.31.0e6c5bf4
Bump husky from 6.0.0 to 7.0.1f781bc4
Bump ws from 7.3.1 to 7.5.3Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Bumps shell-quote from 1.7.2 to 1.7.3.
Sourced from shell-quote's changelog.
1.7.3
- Fix a security issue where the regex for windows drive letters allowed some shell meta-characters to escape the quoting rules. (CVE-2021-42740)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Bumps async from 2.6.3 to 2.6.4.
This version was pushed to npm by hargasinski, a new releaser for async since your current version.
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.
Bumps node-fetch from 2.6.0 to 2.6.7.
Sourced from node-fetch's releases.
v2.6.7
Security patch release
Recommended to upgrade, to not leak sensitive cookie and authentication header information to 3th party host while a redirect occurred
What's Changed
- fix: don't forward secure headers to 3th party by
@jimmywarting
in node-fetch/node-fetch#1453Full Changelog: https://github.com/node-fetch/node-fetch/compare/v2.6.6...v2.6.7
v2.6.6
What's Changed
- fix(URL): prefer built in URL version when available and fallback to whatwg by
@jimmywarting
in node-fetch/node-fetch#1352Full Changelog: https://github.com/node-fetch/node-fetch/compare/v2.6.5...v2.6.6
v2.6.2
fixed main path in package.json
v2.6.1
This is an important security release. It is strongly recommended to update as soon as possible.
See CHANGELOG for details.
Sourced from node-fetch's changelog.
Changelog
All notable changes will be recorded here.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
What's Changed
- core: update fetch-blob by
@jimmywarting
in node-fetch/node-fetch#1371- docs: Fix typo around sending a file by
@jimmywarting
in node-fetch/node-fetch#1381- core: (http.request): Cast URL to string before sending it to NodeJS core by
@jimmywarting
in node-fetch/node-fetch#1378- core: handle errors from the request body stream by
@mdmitry01
in node-fetch/node-fetch#1392- core: Better handle wrong redirect header in a response by
@tasinet
in node-fetch/node-fetch#1387- core: Don't use buffer to make a blob by
@jimmywarting
in node-fetch/node-fetch#1402- docs: update readme for TS
@types/node-fetch
by@adamellsworth
in node-fetch/node-fetch#1405- core: Fix logical operator priority to disallow GET/HEAD with non-empty body by
@maxshirshin
in node-fetch/node-fetch#1369- core: Don't use global buffer by
@jimmywarting
in node-fetch/node-fetch#1422- ci: fix main branch by
@dnalborczyk
in node-fetch/node-fetch#1429- core: use more node: protocol imports by
@dnalborczyk
in node-fetch/node-fetch#1428- core: Warn when using data by
@jimmywarting
in node-fetch/node-fetch#1421- docs: Create SECURITY.md by
@JamieSlome
in node-fetch/node-fetch#1445- core: don't forward secure headers to 3th party by
@jimmywarting
in node-fetch/node-fetch#1449New Contributors
@mdmitry01
made their first contribution in node-fetch/node-fetch#1392@tasinet
made their first contribution in node-fetch/node-fetch#1387@adamellsworth
made their first contribution in node-fetch/node-fetch#1405@maxshirshin
made their first contribution in node-fetch/node-fetch#1369@JamieSlome
made their first contribution in node-fetch/node-fetch#1445Full Changelog: https://github.com/node-fetch/node-fetch/compare/v3.1.0...v3.1.2
3.1.0
What's Changed
- fix(Body): Discourage form-data and buffer() by
@jimmywarting
in node-fetch/node-fetch#1212- fix: Pass url string to http.request by
@serverwentdown
in node-fetch/node-fetch#1268- Fix octocat image link by
@lakuapik
in node-fetch/node-fetch#1281- fix(Body.body): Normalize
Body.body
into anode:stream
by@jimmywarting
in node-fetch/node-fetch#924- docs(Headers): Add default Host request header to README.md file by
@robertoaceves
in node-fetch/node-fetch#1316- Update CHANGELOG.md by
@jimmywarting
in node-fetch/node-fetch#1292- Add highWaterMark to cloned properties by
@davesidious
in node-fetch/node-fetch#1162- Update README.md to fix HTTPResponseError by
@thedanfernandez
in node-fetch/node-fetch#1135- docs: switch
url
toURL
by@dhritzkiv
in node-fetch/node-fetch#1318- fix(types): declare buffer() deprecated by
@dnalborczyk
in node-fetch/node-fetch#1345- chore: fix lint by
@dnalborczyk
in node-fetch/node-fetch#1348- refactor: use node: prefix for imports by
@dnalborczyk
in node-fetch/node-fetch#1346- Bump data-uri-to-buffer from 3.0.1 to 4.0.0 by
@dependabot
in node-fetch/node-fetch#1319- Bump mocha from 8.4.0 to 9.1.3 by
@dependabot
in node-fetch/node-fetch#1339- Referrer and Referrer Policy by
@tekwiz
in node-fetch/node-fetch#1057- Add typing for Response.redirect(url, status) by
@c-w
in node-fetch/node-fetch#1169
... (truncated)
1ef4b56
backport of #1449 (#1453)8fe5c4e
2.x: Specify encoding as an optional peer dependency in package.json (#1310)f56b0c6
fix(URL): prefer built in URL version when available and fallback to whatwg (...b5417ae
fix: import whatwg-url in a way compatible with ESM Node (#1303)18193c5
fix v2.6.3 that did not sending query params (#1301)ace7536
fix: properly encode url with unicode characters (#1291)152214c
Fix(package.json): Corrected main file path in package.json (#1274)b5e2e41
update version number2358a6c
Honor the size
option after following a redirect and revert data uri support8c197f8
docs: Fix typos and grammatical errors in README.md (#686)This version was pushed to npm by endless, a new releaser for node-fetch since your current version.
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase
.