Skip to content

GitLab

T
tangled
Commit c8fb7466 authored by Lei Shi's avatar Lei Shi Committed by Chromium LUCI CQ
Browse files
Options

Address the issue of invalidated nodes in Select to Speak 

When a user resize or change a webpage, the selected inline nodes may be
invalidated. To address this issue, we reuse the prior STS logic, which
converts all selected content into node groups. Then, we send the node
groups to TTS one by one. The only entry point for reading new content
will be startSpeechQueue_, which uses updateNodeGroups_ to convert nodes
into node groups, and start TTS using startCurrentNodeGroup_.

This CL also refactored the names of variables and removed unnecessary
variables. The prior CLs (http://crrev.com/c/2582524,
http://crrev.com/c/2587799, and http://crrev.com/c/2587820) have
modified navigation features so that they do not rely on
navigationState_ anymore.Thus I implement the following rename and
refactor:
1. Rename this.currentNode_ to this.currentNodeGroupItem_ since the
variable is actually for a node group item.
2. Rename this.currentNodeGroupIndex_ to this.currentNodeGroupItemIndex_
since the variable is meant to be the index for node group item.
3. Add this.currentNodeGroups_ and this.currentNodeGroupIndex_, which
are used for tracking the selected node groups and the current speaking
node group.
4. Refactor this.navigationState_.currentCharIndex and
this.navigationState_.isUserSelectContent. to this.currentCharIndex_ and
this.isUserSelectContent_ to keep code concise.

Also, we added a test for the case of invalidated nodes. Test credits to
joelriley@google.com.

Bug: 1160004
Change-Id: I92e0a1acc25265df476ea248be39d05ec06726d6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2587565
Commit-Queue: Lei Shi <leileilei@google.com>
Reviewed-by: default avatarDavid Tseng <dtseng@chromium.org>
Cr-Commit-Position: refs/heads/master@{#839018}
parent 6de88b2e
Show whitespace changes
Inline Side-by-side
Showing with 314 additions and 273 deletions
chrome/browser/resources/chromeos/accessibility/select_to_speak/select_to_speak.js
View file @ c8fb7466
......@@ -128,81 +128,69 @@ class SelectToSpeak {
/** @private {boolean} */
this.readAfterClose_ = true;
/** @private {?ParagraphUtils.NodeGroupItem} */
this.currentNode_ = null;
/** @private {number} */
this.currentNodeGroupIndex_ = -1;
/** @private {?ParagraphUtils.NodeGroup} */
this.currentNodeGroup_ = null;
/**
* The indexes within the current node representing the word currently being
* spoken. Only updated if word highlighting is enabled.
* @private {?Object}
*/
this.currentNodeWord_ = null;
/**
* These navigation state variables enable us to separate the
* functionalities of node enqueueing, navigation control, and TTS playback.
* @private {!{currentNodes:!Array<AutomationNode>,
* currentNodeGroupStartNodeIndex: number,
* currentCharIndex: number,
* currentStartCharIndex: (number|undefined),
* currentEndCharIndex: (number|undefined),
* supportsNavigationPanel: boolean}}
* isUserSelectedContent: boolean}}
*/
this.navigationState_ = {
/**
* The current enqueued Automation nodes.
* The node groups to be spoken. We process content into node groups and
* pass one node group at a time to the TTS engine. Note that we do not use
* node groups for user-selected text in Gsuite. See readNodesInSelection_.
* @private {!Array<!ParagraphUtils.NodeGroup>}
*/
currentNodes: [],
this.currentNodeGroups_ = [];
/**
* The index indicating which node in |currentNodes| is used as the start
* of the current NodeGroup. STS will select a few nodes from
* |currentNodes| to form a NodeGroup, which is the basic unit for a
* TTS utterance. A NodeGroup could be a paragraph and a block of text.
* This number is used to generate a NodeGroup from |currentNodes|.
* The index for the node group currently being spoken in
* |this.currentNodeGroups_|.
* @private {number}
*/
currentNodeGroupStartNodeIndex: -1,
this.currentNodeGroupIndex_ = -1;
/**
* The start char index of the word to be spoken. The index is relative
* to the NodeGroup.
* The node group item currently being spoken. A node group item is a
* representation of the original input nodes, but may not be the same. For
* example, an input inline text node will be represented by its static text
* node in the node group item.
* @private {?ParagraphUtils.NodeGroupItem}
*/
currentCharIndex: -1,
this.currentNodeGroupItem_ = null;
/**
* The current start offset. The index is relative to the Automation node.
* When a user selects text, they could start from the middle of a node.
* This variable indicates where the user selection starts from within
* the first node of |currentNodes|. If set to undefined, we will use the
* entire first node.
* The index for the current node group item within the current node group,
* The current node group can be accessed from |this.currentNodeGroups_|
* using |this.currentNodeGroupIndex_|. In most cases,
* |this.currentNodeGroupItemIndex_| can be used to get
* |this.currentNodeGroupItem_| from the current node group. However, in
* Gsuite, we will have node group items outside of a node group.
* @private {number}
*/
currentStartCharIndex: undefined,
this.currentNodeGroupItemIndex_ = -1;
/**
* The current end offset - see currentStartCharIndex, above.
* The indexes within the current node group item representing the word
* currently being spoken. Only updated if word highlighting is enabled.
* @private {?Object}
*/
currentEndCharIndex: undefined,
this.currentNodeWord_ = null;
/**
* Whether the current nodes support use of the navigation panel.
* The start char index of the word to be spoken. The index is relative
* to the text content of the current node group.
* @private {number}
*/
supportsNavigationPanel: true,
this.currentCharIndex_ = -1;
/**
* Whether we are reading user-selected content. True if the current
* content is from mouse or keyboard selection. False if the current
* content is processed by the navigation features like paragraph
* navigation, sentence navigation, pause and resume.
* @private {boolean}
*/
isUserSelectedContent: false,
};
this.isUserSelectedContent_ = false;
/**
* Whether the current nodes support use of the navigation panel.
* @private {boolean}
*/
this.supportsNavigationPanel_ = true;
/**
* The position of the current focus ring, which usually highlights the
......@@ -212,9 +200,6 @@ class SelectToSpeak {
*/
this.currentFocusRing_ = [];
/** @private {?AutomationNode} */
this.currentBlockParent_ = null;
/** @private {boolean} */
this.visible_ = true;
......@@ -282,6 +267,14 @@ class SelectToSpeak {
this.overrideSpeechRate_ = null;
}
/**
* Gets the node group currently being spoken.
* @return {!ParagraphUtils.NodeGroup|undefined}
*/
getCurrentNodeGroup_() {
return this.currentNodeGroups_[this.currentNodeGroupIndex_];
}
/**
* Determines if navigation controls should be shown (and other related
* functionality, such as auto-dismiss and click-to-navigate to sentence,
......@@ -291,7 +284,7 @@ class SelectToSpeak {
shouldShowNavigationControls_() {
return this.navigationControlFlag_ &&
this.prefsManager_.navigationControlsEnabled() &&
this.navigationState_.supportsNavigationPanel;
this.supportsNavigationPanel_;
}
/**
......@@ -520,6 +513,8 @@ class SelectToSpeak {
MetricsUtils.recordStartEvent(
MetricsUtils.StartSpeechMethod.KEYSTROKE, this.prefsManager_);
} else {
// Gsuite apps include webapps beyond Docs, see getGSuiteAppRoot and
// GSUITE_APP_REGEXP.
const gsuiteAppRootNode = getGSuiteAppRoot(focusedNode);
if (!gsuiteAppRootNode) {
return;
......@@ -532,7 +527,7 @@ class SelectToSpeak {
}
const tab = tabs[0];
this.inputHandler_.onRequestReadClipboardData();
this.currentNode_ =
this.currentNodeGroupItem_ =
new ParagraphUtils.NodeGroupItem(gsuiteAppRootNode, 0, false);
chrome.tabs.executeScript(tab.id, {
allFrames: true,
......@@ -648,9 +643,9 @@ class SelectToSpeak {
* |this.ttsPaused_| was true while tts is speaking. This function also sets
* the |this.pauseCompleteCallback_|, which will be executed at the end of
* the pause process in |updatePauseStatusFromTtsEvent_|. This enables us to
* execute functions when the pause request is finished. For example, we can
* adjust |this.navigationState_| after a pause is fulfilled, and then resume
* to navigate to different positions.
* execute functions when the pause request is finished. For example, to
* navigate the next sentence, we trigger pause_ and start finding the next
* sentence when the pause function is fulfilled.
* @return {!Promise}
* @private
*/
......@@ -666,7 +661,7 @@ class SelectToSpeak {
// user-selected content. This enables us to distinguish between a user-
// trigger pause from the auto pause happening at the end of user-selected
// content.
this.navigationState_.isUserSelectedContent = false;
this.isUserSelectedContent_ = false;
});
}
......@@ -684,15 +679,16 @@ class SelectToSpeak {
if (!this.isPaused_()) {
return;
}
const currentNodeGroup = this.getCurrentNodeGroup_();
// If there is no processed node group, that means the user has not selected
// anything. Ignore the resume command.
if (!this.currentNodeGroup_) {
if (!currentNodeGroup) {
return;
}
const {nodes: remainingNodes, offset} =
NodeUtils.getNextNodesInParagraphFromNodeGroup(
this.currentNodeGroup_, this.navigationState_.currentCharIndex,
constants.Dir.FORWARD);
currentNodeGroup, this.currentCharIndex_, constants.Dir.FORWARD);
// There is no remaining nodes in this paragraph so we navigate to the next
// paragraph.
if (remainingNodes.length === 0) {
......@@ -700,9 +696,9 @@ class SelectToSpeak {
return;
}
if (this.navigationState_.isUserSelectedContent ||
if (this.isUserSelectedContent_ ||
SentenceUtils.isSentenceStart(
this.currentNodeGroup_, this.navigationState_.currentCharIndex)) {
currentNodeGroup, this.currentCharIndex_)) {
// If we are resuming from the end of user-selected content or if we are
// at the start of the current sentence, we should start reading the
// remaining content.
......@@ -728,27 +724,10 @@ class SelectToSpeak {
stopAll_() {
chrome.tts.stop();
this.clearFocusRing_();
this.clearNavigationStateVariables_();
this.overrideSpeechRate_ = null; // Reset speech rate to system default
this.onStateChanged_(SelectToSpeakState.INACTIVE);
}
/**
* Clears the member variables for the navigation state.
* @private
*/
clearNavigationStateVariables_() {
this.navigationState_ = {
currentNodes: [],
currentNodeGroupStartNodeIndex: -1,
currentCharIndex: -1,
currentStartCharIndex: undefined,
currentEndCharIndex: undefined,
supportsNavigationPanel: true,
isUserSelectedContent: false,
};
}
/**
* Clears the current focus ring and node, but does
* not stop the speech.
......@@ -757,14 +736,27 @@ class SelectToSpeak {
clearFocusRingAndNode_() {
this.clearFocusRing_();
// Clear the node and also stop the interval testing.
this.currentNode_ = null;
this.currentNodeGroupIndex_ = -1;
this.currentNodeWord_ = null;
this.resetNodes_();
this.supportsNavigationPanel_ = true;
this.isUserSelectedContent_ = false;
clearInterval(this.intervalId_);
this.intervalId_ = undefined;
this.scrollToSpokenNode_ = false;
}
/**
* Resets the instance variables for nodes and node groups.
* @private
*/
resetNodes_() {
this.currentNodeGroups_ = [];
this.currentNodeGroupIndex_ = -1;
this.currentNodeGroupItem_ = null;
this.currentNodeGroupItemIndex_ = -1;
this.currentNodeWord_ = null;
this.currentCharIndex_ = -1;
}
/**
* Update the navigation floating panel.
* @private
......@@ -1037,9 +1029,11 @@ class SelectToSpeak {
* @private
*/
async navigateToNextSentence_(direction, skipCurrentSentence = false) {
const currentNodeGroup = this.getCurrentNodeGroup_();
// An empty node group is not expected and means that the user has not
// enqueued any text.
if (!this.currentNodeGroup_) {
if (!currentNodeGroup) {
return;
}
......@@ -1050,8 +1044,8 @@ class SelectToSpeak {
// Checks the next sentence within this node group. If we have enqueued the
// next sentence that fulfilled the requirements, return.
if (this.enqueueNextSentenceWithinNodeGroup_(
this.currentNodeGroup_, this.navigationState_.currentCharIndex,
direction, skipCurrentSentence)) {
currentNodeGroup, this.currentCharIndex_, direction,
skipCurrentSentence)) {
return;
}
......@@ -1063,8 +1057,7 @@ class SelectToSpeak {
// searching backward, the offset is the char index pointing to the char
// after the remaining content.
const {nodes, offset} = NodeUtils.getNextNodesInParagraphFromNodeGroup(
this.currentNodeGroup_, this.navigationState_.currentCharIndex,
direction);
currentNodeGroup, this.currentCharIndex_, direction);
// If we have reached to the end of a paragraph, enqueue the sentence from
// the next paragraph.
if (nodes.length === 0) {
......@@ -1090,12 +1083,12 @@ class SelectToSpeak {
}
// When searching backward, we need to adjust |skipCurrentSentence| if it
// is true. The remaining content we get excludes the char at
// |this.navigationState_.currentCharIndex|. If this char is a sentence
// |this.currentCharIndex_|. If this char is a sentence
// start, we have already skipped the current sentence so we need to change
// |skipCurrentSentence| to false for the next search.
if (direction === constants.Dir.BACKWARD && skipCurrentSentence) {
const currentPositionIsSentenceStart = SentenceUtils.isSentenceStart(
this.currentNodeGroup_, this.navigationState_.currentCharIndex);
currentNodeGroup, this.currentCharIndex_);
if (currentPositionIsSentenceStart) {
skipCurrentSentence = false;
}
......@@ -1264,12 +1257,16 @@ class SelectToSpeak {
locateNodesForNextParagraph_(direction) {
// Use current block parent as starting point to navigate from. If it is not
// a valid block, then use one of the nodes that are currently activated.
let node = this.currentBlockParent_;
if ((node === null || node.isRootNode) && this.currentNodeGroup_ &&
this.currentNodeGroup_.nodes.length > 0) {
node = this.currentNodeGroup_.nodes[0].node;
const currentNodeGroup = this.getCurrentNodeGroup_();
if (!currentNodeGroup) {
return [];
}
if (node === null) {
let node = currentNodeGroup.blockParent;
if ((node === null || node.isRootNode || node.role === undefined) &&
currentNodeGroup.nodes.length > 0) {
node = currentNodeGroup.nodes[0].node;
}
if (node === null || node.role === undefined) {
// Could not find any nodes to navigate from.
return [];
}
......@@ -1333,8 +1330,7 @@ class SelectToSpeak {
/**
* Enqueue nodes to TTS queue and start TTS. This function can be used for
* adding nodes, either from user selection (e.g., mouse selection) or
* navigation control (e.g., next paragraph). This function will overwrite
* |this.navigationState_| and start TTS according to the offsets.
* navigation control (e.g., next paragraph).
* @param {!Array<AutomationNode>} nodes The nodes to speak.
* @param {!{clearFocusRing: (boolean|undefined),
* startCharIndex: (number|undefined),
......@@ -1357,7 +1353,7 @@ class SelectToSpeak {
const clearFocusRing = params.clearFocusRing || false;
let startCharIndex = params.startCharIndex;
let endCharIndex = params.endCharIndex;
const isUserSelectedContent = params.isUserSelectedContent || false;
this.isUserSelectedContent_ = params.isUserSelectedContent || false;
this.prepareForSpeech_(clearFocusRing /* clear the focus ring */);
......@@ -1380,101 +1376,96 @@ class SelectToSpeak {
endCharIndex = undefined;
}
// Update the navigation state variables.
this.navigationState_ = {
currentNodes: nodes,
currentNodeGroupStartNodeIndex: 0,
currentCharIndex: 0,
currentStartCharIndex: startCharIndex,
currentEndCharIndex: endCharIndex,
supportsNavigationPanel: this.isNavigationPanelSupported_(nodes),
isUserSelectedContent,
};
this.supportsNavigationPanel_ = this.isNavigationPanelSupported_(nodes);
this.updateNodeGroups_(nodes, startCharIndex, endCharIndex);
// Play TTS according to the current state variables.
this.startCurrentNodeGroup_();
}
/**
* Start TTS according to the five variables in |this.navigationState_|. This
* function will first construct a NodeGroup based on |currentNodes| and
* |currentNodeGroupStartNodeIndex|. Then, it will clip texts based on
* |currentCharIndex|, |currentStartCharIndex|, and |currentEndCharIndex|.
* Lastly, it will start TTS using the processed text.
* Updates the node groups to be spoken. Converts |nodes|, |startCharIndex|,
* and |endCharIndex| into node groups, and updates |this.currentNodeGroups_|
* and |this.currentNodeGroupIndex_|.
* @param {!Array<AutomationNode>} nodes The nodes to speak.
* @param {number=} startCharIndex The index into the first node's text at
* which to start speaking. If this is not passed, will start at 0.
* @param {number=} endCharIndex The index into the last node's text at which
* to end speech. If this is not passed, will stop at the end.
* @private
*/
startCurrentNodeGroup_() {
const currentNodes = this.navigationState_.currentNodes;
const currentNodeGroupStartNodeIndex =
this.navigationState_.currentNodeGroupStartNodeIndex;
const currentCharIndex = this.navigationState_.currentCharIndex;
const currentStartCharIndex = this.navigationState_.currentStartCharIndex;
const currentEndCharIndex = this.navigationState_.currentEndCharIndex;
// Reaches to the end of the current nodes.
if (currentNodeGroupStartNodeIndex >= currentNodes.length) {
return;
}
// When navigation controls are enabled, disable the
// clipping of overflow words. When overflow words are clipped, words
// scrolled out of view are clipped, which is undesirable for our navigation
// features as we generate node groups for next/previous paragraphs which
// may be fully or partially scrolled out of view.
const nodeGroup = ParagraphUtils.buildNodeGroup(
currentNodes, currentNodeGroupStartNodeIndex, {
updateNodeGroups_(nodes, startCharIndex, endCharIndex) {
this.resetNodes_();
for (let i = 0; i < nodes.length; i++) {
// When navigation controls are enabled, disable the clipping of overflow
// words. When overflow words are clipped, words scrolled out of view are
// clipped, which is undesirable for our navigation features as we
// generate node groups for next/previous paragraphs which may be fully or
// partially scrolled out of view.
const nodeGroup = ParagraphUtils.buildNodeGroup(nodes, i, {
splitOnLanguage: this.enableLanguageDetectionIntegration_,
clipOverflowWords: !this.shouldShowNavigationControls_(),
});
// |currentCharIndex| is the start char index of the word to be spoken in
// the nodeGroup text. If the |currentCharIndex| is non-zero, that means we
// are resuming from prior TTS. We trim the text regardless which NodeGroup
// we are, as a user could possibly pause at any NodeGroup.
this.applyOffset(nodeGroup, currentCharIndex, true /* isStartOffset */);
const isFirstNodeGroup = currentNodeGroupStartNodeIndex === 0;
const isFirstNodeGroup = i === 0;
const shouldApplyStartOffset =
isFirstNodeGroup && currentStartCharIndex !== undefined;
isFirstNodeGroup && startCharIndex !== undefined;
const firstNodeHasInlineText =
nodeGroup.nodes.length > 0 && nodeGroup.nodes[0].hasInlineText;
if (shouldApplyStartOffset && firstNodeHasInlineText) {
// We assume that the start offset will only be applied to the first node
// in the first NodeGroup. The |currentStartCharIndex| needs to be
// We assume that the start offset will only be applied to the first
// node in the first NodeGroup. The |startCharIndex| needs to be
// adjusted. The first node of the NodeGroup may not be at the beginning
// of the parent of the NodeGroup. (e.g., an inlineText in its staticText
// parent). Thus, we need to adjust the start index.
// of the parent of the NodeGroup. (e.g., an inlineText in its
// staticText parent). Thus, we need to adjust the start index.
const startIndexInNodeParent =
ParagraphUtils.getStartCharIndexInParent(currentNodes[0]);
const startIndexInNodeGroup = currentStartCharIndex +
startIndexInNodeParent + nodeGroup.nodes[0].startChar;
ParagraphUtils.getStartCharIndexInParent(nodes[0]);
const startIndexInNodeGroup = startCharIndex + startIndexInNodeParent +
nodeGroup.nodes[0].startChar;
this.applyOffset(
nodeGroup, startIndexInNodeGroup, true /* isStartOffset */);
}
const isLastNodeGroup = (nodeGroup.endIndex === currentNodes.length - 1);
// Advance i to the end of this group, to skip all nodes it contains.
i = nodeGroup.endIndex;
const isLastNodeGroup = (i === nodes.length - 1);
const shouldApplyEndOffset =
isLastNodeGroup && currentEndCharIndex !== undefined;
isLastNodeGroup && endCharIndex !== undefined;
const lastNodeHasInlineText = nodeGroup.nodes.length > 0 &&
nodeGroup.nodes[nodeGroup.nodes.length - 1].hasInlineText;
if (shouldApplyEndOffset && lastNodeHasInlineText) {
// We assume that the end offset will only be applied to the last node in
// the last NodeGroup. Similarly, |currentEndCharIndex| needs to be
// We assume that the end offset will only be applied to the last node
// in the last NodeGroup. Similarly, |endCharIndex| needs to be
// adjusted.
const startIndexInNodeParent = ParagraphUtils.getStartCharIndexInParent(
currentNodes[nodeGroup.endIndex]);
const endIndexInNodeGroup = currentEndCharIndex + startIndexInNodeParent +
const startIndexInNodeParent =
ParagraphUtils.getStartCharIndexInParent(nodes[i]);
const endIndexInNodeGroup = endCharIndex + startIndexInNodeParent +
nodeGroup.nodes[nodeGroup.nodes.length - 1].startChar;
this.applyOffset(
nodeGroup, endIndexInNodeGroup, false /* isStartOffset */);
}
if (nodeGroup.nodes.length === 0 && !isLastNodeGroup) {
// If the current nodeGroup is empty, we end this node group early, as if
// we have completed this node group.
this.onNodeGroupSpeakingCompleted_(
currentNodeGroupStartNodeIndex /* currentNodeGroupEndIndex */);
continue;
}
this.currentNodeGroups_.push(nodeGroup);
}
// Sets the initial node group index to zero if this.currentNodeGroups_ has
// items.
if (this.currentNodeGroups_.length > 0) {
this.currentNodeGroupIndex_ = 0;
}
}
/**
* Starts reading the current node group.
* @private
*/
startCurrentNodeGroup_() {
const nodeGroup = this.getCurrentNodeGroup_();
if (!nodeGroup) {
return;
}
const options = {};
// Copy options so we can add lang below
Object.assign(options, this.prefsManager_.speechOptions());
......@@ -1491,30 +1482,24 @@ class SelectToSpeak {
options.onEvent = (event) => {
if (event.type === 'start' && nodeGroup.nodes.length > 0) {
this.updatePauseStatusFromTtsEvent_(false /* shouldPause */);
// TODO(leileilei): We can get rid of this.currentBlockParent_ if it is
// always equal to currentNodeGroup_.blockParent.
this.currentBlockParent_ = nodeGroup.blockParent;
this.currentNodeGroup_ = nodeGroup;
// Update |currentCharIndex|. Find the first non-space char index in
// nodeGroup text, or 0 if the text is undefined or the first char is
// Update |this.currentCharIndex_|. Find the first non-space char index
// in nodeGroup text, or 0 if the text is undefined or the first char is
// non-space.
this.navigationState_.currentCharIndex = nodeGroupText.search(/\S|$/);
this.currentCharIndex_ = nodeGroupText.search(/\S|$/);
this.syncCurrentNodeWithCharIndex_(
nodeGroup, this.navigationState_.currentCharIndex /* charIndex */);
this.syncCurrentNodeWithCharIndex_(nodeGroup, this.currentCharIndex_);
if (this.prefsManager_.wordHighlightingEnabled()) {
// At 'start', find the first word and highlight that. Clear the
// previous word in the node.
this.currentNodeWord_ = null;
// If |currentCharIndex| is not 0, that means we have applied a start
// offset. Thus, we need to pass startIndexInNodeGroup to
// If |this.currentCharIndex_| is not 0, that means we have applied a
// start offset. Thus, we need to pass startIndexInNodeGroup to
// opt_startIndex and overwrite the word boundaries in the original
// node.
this.updateNodeHighlight_(
nodeGroupText, this.navigationState_.currentCharIndex,
this.navigationState_.currentCharIndex !== 0 ?
this.navigationState_.currentCharIndex :
nodeGroupText, this.currentCharIndex_,
this.currentCharIndex_ !== 0 ? this.currentCharIndex_ :
undefined);
} else {
this.testCurrentNode_();
......@@ -1540,9 +1525,13 @@ class SelectToSpeak {
this.updatePauseStatusFromTtsEvent_(true /* shouldPause */);
}
} else if (event.type === 'end') {
this.onNodeGroupSpeakingCompleted_(
nodeGroup.endIndex /* currentNodeGroupEndIndex */);
this.onNodeGroupSpeakingCompleted_();
} else if (event.type === 'word') {
// The Closure compiler doesn't realize that we did a !nodeGroup earlier
// so we check again here.
if (!nodeGroup) {
return;
}
this.onTtsWordEvent_(event, nodeGroup);
}
};
......@@ -1554,20 +1543,18 @@ class SelectToSpeak {
* indicated by the end index. If we have reached the last node group, this
* function will update STS status depending whether the navigation feature is
* enabled.
* @param {number} currentNodeGroupEndIndex the index of the last node in the
* current node group. The index is relative to
* |this.navigationState_.currentNodes|.
* @private
*/
onNodeGroupSpeakingCompleted_(currentNodeGroupEndIndex) {
onNodeGroupSpeakingCompleted_() {
const currentNodeGroup = this.getCurrentNodeGroup_();
// Update the current char index to the end of the text content in this
// nodeGroup.
const nodeGroupText =
(this.currentNodeGroup_ && this.currentNodeGroup_.text) || '';
this.navigationState_.currentCharIndex = nodeGroupText.trimEnd().length;
const nodeGroupText = (currentNodeGroup && currentNodeGroup.text) || '';
this.currentCharIndex_ = nodeGroupText.trimEnd().length;
const isLastNodeGroup =
(currentNodeGroupEndIndex ===
this.navigationState_.currentNodes.length - 1);
(this.currentNodeGroupIndex_ === this.currentNodeGroups_.length - 1);
if (isLastNodeGroup) {
if (!this.shouldShowNavigationControls_()) {
this.onStateChanged_(SelectToSpeakState.INACTIVE);
......@@ -1578,18 +1565,14 @@ class SelectToSpeak {
return;
}
// Navigate to the next NodeGroup. Don't change |currentNodes|,
// |currentStartCharIndex|, or |currentEndCharIndex|.
this.navigationState_.currentCharIndex = 0;
this.navigationState_.currentNodeGroupStartNodeIndex =
currentNodeGroupEndIndex + 1;
// Play TTS.
// Start reading the next node group.
this.currentNodeGroupIndex_++;
this.startCurrentNodeGroup_();
}
/**
* Update |this.currentNode_|, the current speaking or the node to be spoken
* in the node group.
* Update |this.currentNodeGroupItem_|, the current speaking or the node to be
* spoken in the node group.
* @param {ParagraphUtils.NodeGroup} nodeGroup the current nodeGroup.
* @param {number} charIndex the start char index of the word to be spoken.
* The index is relative to the entire NodeGroup.
......@@ -1604,19 +1587,22 @@ class SelectToSpeak {
opt_startFromNodeGroupIndex = 0;
}
// There is no speaking word, set the NodeGroupIndex to 0.
// There is no speaking word, set the NodeGroupItemIndex to 0.
if (charIndex <= 0) {
this.currentNodeGroupIndex_ = 0;
this.currentNode_ = nodeGroup.nodes[this.currentNodeGroupIndex_];
return this.currentNodeGroupIndex_ === opt_startFromNodeGroupIndex;
this.currentNodeGroupItemIndex_ = 0;
this.currentNodeGroupItem_ =
nodeGroup.nodes[this.currentNodeGroupItemIndex_];
return this.currentNodeGroupItemIndex_ === opt_startFromNodeGroupIndex;
}
// Sets the this.currentNodeGroupIndex_ to |opt_startFromNodeGroupIndex|
this.currentNodeGroupIndex_ = opt_startFromNodeGroupIndex;
this.currentNode_ = nodeGroup.nodes[this.currentNodeGroupIndex_];
// Sets the |this.currentNodeGroupItemIndex_| to
// |opt_startFromNodeGroupIndex|
this.currentNodeGroupItemIndex_ = opt_startFromNodeGroupIndex;
this.currentNodeGroupItem_ =
nodeGroup.nodes[this.currentNodeGroupItemIndex_];
if (this.currentNodeGroupIndex_ + 1 < nodeGroup.nodes.length) {
let next = nodeGroup.nodes[this.currentNodeGroupIndex_ + 1];
if (this.currentNodeGroupItemIndex_ + 1 < nodeGroup.nodes.length) {
let next = nodeGroup.nodes[this.currentNodeGroupItemIndex_ + 1];
let nodeUpdated = false;
// TODO(katie): For something like a date, the start and end
// node group nodes can actually be different. Example:
......@@ -1626,7 +1612,7 @@ class SelectToSpeak {
// start char index of the target word, we just need to make sure the
// next.startchar is bigger than it.
while (next && charIndex >= next.startChar &&
this.currentNodeGroupIndex_ + 1 < nodeGroup.nodes.length) {
this.currentNodeGroupItemIndex_ + 1 < nodeGroup.nodes.length) {
next = this.incrementCurrentNodeAndGetNext_(nodeGroup);
nodeUpdated = true;
}
......@@ -1681,10 +1667,9 @@ class SelectToSpeak {
onTtsWordEvent_(event, nodeGroup) {
// Not all speech engines include length in the ttsEvent object. .
const hasLength = event.length !== undefined && event.length >= 0;
// Only update the currentCharIndex if event has a higher charIndex. TTS
// sometimes will report an incorrect number at the end of an utterance.
this.navigationState_.currentCharIndex =
Math.max(event.charIndex, this.navigationState_.currentCharIndex);
// Only update the |this.currentCharIndex_| if event has a higher charIndex.
// TTS sometimes will report an incorrect number at the end of an utterance.
this.currentCharIndex_ = Math.max(event.charIndex, this.currentCharIndex_);
console.debug(nodeGroup.text + ' (index ' + event.charIndex + ')');
let debug = '-'.repeat(event.charIndex);
if (hasLength) {
......@@ -1695,10 +1680,10 @@ class SelectToSpeak {
console.debug(debug);
// First determine which node contains the word currently being spoken,
// and update this.currentNode_, this.currentNodeWord_, and
// this.currentNodeGroupIndex_ to match.
// and update this.currentNodeGroupItem_, this.currentNodeWord_, and
// this.currentNodeGroupItemIndex_ to match.
const nodeUpdated = this.syncCurrentNodeWithCharIndex_(
nodeGroup, event.charIndex, this.currentNodeGroupIndex_);
nodeGroup, event.charIndex, this.currentNodeGroupItemIndex_);
if (nodeUpdated) {
if (!this.prefsManager_.wordHighlightingEnabled()) {
// If we are doing a per-word highlight, we will test the
......@@ -1712,8 +1697,9 @@ class SelectToSpeak {
if (this.prefsManager_.wordHighlightingEnabled()) {
if (hasLength) {
this.currentNodeWord_ = {
'start': event.charIndex - this.currentNode_.startChar,
'end': event.charIndex + event.length - this.currentNode_.startChar
'start': event.charIndex - this.currentNodeGroupItem_.startChar,
'end': event.charIndex + event.length -
this.currentNodeGroupItem_.startChar
};
this.testCurrentNode_();
} else {
......@@ -1736,15 +1722,16 @@ class SelectToSpeak {
*/
incrementCurrentNodeAndGetNext_(nodeGroup) {
// Move to the next node.
this.currentNodeGroupIndex_ += 1;
this.currentNode_ = nodeGroup.nodes[this.currentNodeGroupIndex_];
this.currentNodeGroupItemIndex_ += 1;
this.currentNodeGroupItem_ =
nodeGroup.nodes[this.currentNodeGroupItemIndex_];
// Setting this.currentNodeWord_ to null signals it should be recalculated
// later.
this.currentNodeWord_ = null;
if (this.currentNodeGroupIndex_ + 1 >= nodeGroup.nodes.length) {
if (this.currentNodeGroupItemIndex_ + 1 >= nodeGroup.nodes.length) {
return null;
}
return nodeGroup.nodes[this.currentNodeGroupIndex_ + 1];
return nodeGroup.nodes[this.currentNodeGroupItemIndex_ + 1];
}
/**
......@@ -1877,9 +1864,13 @@ class SelectToSpeak {
// TODO: Better test: has no siblings in the group, highlight just
// the one node. if it has siblings, highlight the parent.
let focusRingRect;
if (this.currentBlockParent_ !== null &&
node.role === RoleType.INLINE_TEXT_BOX) {
focusRingRect = this.currentBlockParent_.location;
const currentNodeGroup = this.getCurrentNodeGroup_();
if (!currentNodeGroup) {
return;
}
const currentBlockParent = currentNodeGroup.blockParent;
if (currentBlockParent !== null && node.role === RoleType.INLINE_TEXT_BOX) {
focusRingRect = currentBlockParent.location;
} else {
focusRingRect = node.location;
}
......@@ -1892,22 +1883,22 @@ class SelectToSpeak {
* @private
*/
testCurrentNode_() {
if (this.currentNode_ == null) {
if (this.currentNodeGroupItem_ == null) {
return;
}
if (this.currentNode_.node.location === undefined) {
if (this.currentNodeGroupItem_.node.location === undefined) {
// Don't do the hit test because there is no location to test against.
// Just directly update Select To Speak from node state.
this.updateFromNodeState_(this.currentNode_, false);
this.updateFromNodeState_(this.currentNodeGroupItem_, false);
} else {
this.updateHighlightAndFocus_(this.currentNode_);
this.updateHighlightAndFocus_(this.currentNodeGroupItem_);
// Do a hit test to make sure the node is not in a background window
// or minimimized. On the result checkCurrentNodeMatchesHitTest_ will be
// called, and we will use that result plus the currentNode's state to
// determine how to set the focus and whether to stop speech.
this.desktop_.hitTest(
this.currentNode_.node.location.left,
this.currentNode_.node.location.top, EventType.HOVER);
this.currentNodeGroupItem_.node.location.left,
this.currentNodeGroupItem_.node.location.top, EventType.HOVER);
}
}
......@@ -1917,13 +1908,13 @@ class SelectToSpeak {
* @private
*/
onHitTestCheckCurrentNodeMatches_(evt) {
if (this.currentNode_ == null) {
if (this.currentNodeGroupItem_ == null) {
return;
}
chrome.automation.getFocus(function(focusedNode) {
var window = NodeUtils.getNearestContainingWindow(evt.target);
var currentWindow =
NodeUtils.getNearestContainingWindow(this.currentNode_.node);
NodeUtils.getNearestContainingWindow(this.currentNodeGroupItem_.node);
var inForeground =
currentWindow != null && window != null && currentWindow === window;
if (!inForeground &&
......@@ -1943,7 +1934,7 @@ class SelectToSpeak {
NodeUtils.getNearestContainingWindow(focusedNode.root);
inForeground = focusedWindow != null && currentWindow === focusedWindow;
}
this.updateFromNodeState_(this.currentNode_, inForeground);
this.updateFromNodeState_(this.currentNodeGroupItem_, inForeground);
}.bind(this));
}
......@@ -1985,20 +1976,20 @@ class SelectToSpeak {
}
// Get the next word based on the event's charIndex.
const nextWordStart =
WordUtils.getNextWordStart(text, charIndex, this.currentNode_);
WordUtils.getNextWordStart(text, charIndex, this.currentNodeGroupItem_);
// The |WordUtils.getNextWordEnd| will find the correct end based on the
// trimmed text, so there is no need to provide additional input like
// opt_startIndex.
const nextWordEnd = WordUtils.getNextWordEnd(
text, opt_startIndex === undefined ? nextWordStart : opt_startIndex,
this.currentNode_);
this.currentNodeGroupItem_);
// Map the next word into the node's index from the text.
const nodeStart = opt_startIndex === undefined ?
nextWordStart - this.currentNode_.startChar :
opt_startIndex - this.currentNode_.startChar;
nextWordStart - this.currentNodeGroupItem_.startChar :
opt_startIndex - this.currentNodeGroupItem_.startChar;
const nodeEnd = Math.min(
nextWordEnd - this.currentNode_.startChar,
NodeUtils.nameLength(this.currentNode_.node));
nextWordEnd - this.currentNodeGroupItem_.startChar,
NodeUtils.nameLength(this.currentNodeGroupItem_.node));
if ((this.currentNodeWord_ == null ||
nodeStart >= this.currentNodeWord_.end) &&
nodeStart <= nodeEnd) {
......
chrome/browser/resources/chromeos/accessibility/select_to_speak/select_to_speak_navigation_control_test.js
View file @ c8fb7466
......@@ -588,3 +588,53 @@ TEST_F(
this.mockTts.pendingUtterances()[0], '. Sentence two.');
});
});
TEST_F('SelectToSpeakNavigationControlTest', 'ResizeWhilePlaying', function() {
const longLine =
'Second paragraph is longer than 300 pixels and will wrap when resized';
const bodyHtml = `
<script type="text/javascript">
function doResize() {
document.getElementById('resize').style.width = '100px';
}
</script>
<div id="content">
<p>First paragraph</p>
<p id='resize' style='width:300px; font-size: 1em'>
${longLine}
</p>
</div>
<button onclick="doResize()">Resize</button>
`;
this.runWithLoadedTree(
this.generateHtmlWithSelectedElement('content', bodyHtml), (root) => {
this.triggerReadSelectedText();
// Speaks the first paragraph.
assertTrue(this.mockTts.currentlySpeaking());
assertEquals(this.mockTts.pendingUtterances().length, 1);
this.assertEqualsCollapseWhitespace(
this.mockTts.pendingUtterances()[0], 'First paragraph');
const resizeButton =
root.find({role: 'button', attributes: {name: 'Resize'}});
// Wait for click event, at which point the automation tree should
// be updated from the resize.
resizeButton.addEventListener(
EventType.CLICKED, this.newCallback(() => {
// Trigger next node group by completing first TTS request.
this.mockTts.finishPendingUtterance();
// Should still read second paragraph, even though some nodes
// were invalided from the resize.
assertTrue(this.mockTts.currentlySpeaking());
assertEquals(this.mockTts.pendingUtterances().length, 1);
this.assertEqualsCollapseWhitespace(
this.mockTts.pendingUtterances()[0], longLine);
}));
// Perform resize.
resizeButton.doDefault();
});
});
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment