Web page screen recording implementation

After reading the comments, I suddenly realized that I did not explain it in advance. This article can be said to be a research and learning article. It is a set of solutions that I feel is feasible. I will read some similar code libraries that have been open source to supplement my own knowledge. There are some missing details, so you can treat it as a learning text and use it with caution in production environments.

If your application is connected to the web apm system, then you may know that the apm system can help you capture uncaught errors that occur on the page, provide an error stack, and help you locate the BUG. However, sometimes, when you don’t know the specific operation of the user, there is no way to reproduce the bug. At this time, if there is an operation screen recording, you can clearly understand the user’s operation path, thus reproducing the BUG. And repair.

This idea is relatively simple, which is to use canvas to draw web content. The more famous libraries are: html2canvas. The simple principle of this library is:

• Collect all DOM and store them in a queue;
• According to zIndex, the DOM passes certain rules one by one in order, and the DOM and its CSS style are drawn on the Canvas together.

This implementation is relatively complicated, but we can use it directly, so we can get the screenshot of the web page we want.

In order to make the generated video smoother, we need to generate about 25 frames per second, which means we need 25 screenshots. The flow chart of the idea is as follows:

However, this idea has the most fatal flaw: in order to make the video smooth, we need 25 pictures in one second, and one picture is 300KB. When we need a 30-second video, the total size of the pictures is 220M. Such a large network overhead Obviously not.

In order to reduce network overhead, we change our thinking. We record the next step-by-step operations based on the initial page. When we need to play, we apply these operations in order, so that we can see the changes in the page. This idea separates mouse operations and DOM changes:

Mouse changes:

• Listen to the mouseover event and record the clientX and clientY of the mouse.
• During replay, use js to draw a fake mouse and change the position of the mouse according to the coordinate record.

• Take a full snapshot of the page DOM. Including style collection, JS script removal, and marking each current DOM element with an id through certain rules.
• Monitor all events that may affect the interface, such as various mouse events, input events, scroll events, zoom events, etc. Each event records parameters and target elements. The target element can be the id just recorded. Each time Change events can be recorded as an incremental snapshot.
• Send a certain amount of snapshots to the backend.
• Playback based on snapshots and action chains in the background.

Of course, this explanation is relatively brief. Mouse recording is relatively simple. We will not go into details. We will mainly explain the implementation ideas of DOM monitoring.

First of all, you may think that to achieve a full snapshot of the page, you can directly use outerHTML

 const content = document.documentElement.outerHTML;

This simply records all the DOM of the page. You only need to first add the tag id to the DOM, then get the outerHTML, and then remove the JS script.

However, there is a problem here. The DOM recorded using outerHTML will merge two adjacent TextNodes into one node. When we subsequently monitor DOM changes, we will use MutationObserver . At this time, you need a lot of processing to be compatible with this merging of TextNodes. , otherwise you will not be able to locate the target node of the operation during the restore operation.

So, is there any way we can maintain the original structure of the page DOM?

The answer is yes. Here we use Virtual DOM to record the DOM structure, turn documentElement into Virtual DOM, record it, and regenerate the DOM when restoring later.

We only need to care about two Node types here: Node.TEXT_NODE and Node.ELEMENT_NODE . At the same time, it should be noted that the creation of SVG and SVG sub-elements requires the use of API: createElementNS. Therefore, when we record Virtual DOM, we need to pay attention to the namespace record. The above code:

 const SVG_NAMESPACE = 'http://www.w3.org/2000/svg';const XML_NAMESPACES = ['xmlns', 'xmlns:svg', 'xmlns:xlink'];function createVirtualDom(element, isSVG = false) { switch (element.nodeType) { case Node.TEXT_NODE: return createVirtualText(element); case Node.ELEMENT_NODE: return createVirtualElement(element, isSVG || element.tagName.toLowerCase() === 'svg'); default: return null; }}function createVirtualText(element) { const vText = { text: element.nodeValue, type: 'VirtualText', }; if (typeof element.__flow !== 'undefined') { vText.__flow = element.__flow; } return vText;}function createVirtualElement(element, isSVG = false) { const tagName = element.tagName.toLowerCase(); const children = getNodeChildren(element, isSVG ); const { attr, namespace } = getNodeAttributes(element, isSVG); const vElement = { tagName, type: 'VirtualElement', children, attributes: attr, namespace, }; if (typeof element.__flow !== 'undefined') { vElement.__flow = element.__flow; } return vElement;}function getNodeChildren(element, isSVG = false) { const childNodes = element.childNodes ? [...element.childNodes] : []; const children = []; childNodes.forEach((cnode) => { children.push(createVirtualDom(cnode, isSVG)); }); return children.filter(c => !!c);}function getNodeAttributes(element, isSVG = false) { const attributes = element.attributes ? [...element.attributes] : []; const attr = {}; let namespace; attributes.forEach(({ nodeName, nodeValue }) => { attr[nodeName] = nodeValue; if (XML_NAMESPACES.includes(nodeName)) { namespace = nodeValue; } else if (isSVG) { namespace = SVG_NAMESPACE; } }); return { attr, namespace };}

Through the above code, we can convert the entire documentElement into Virtual DOM, in which __flow is used to record some parameters, including tag ID, etc. Virtual Node records: type, attributes, children, namespace.

It is relatively simple to restore Virtual DOM to DOM. You only need to create the DOM recursively. The nodeFilter is used to filter script elements because we do not need the execution of JS scripts.

 function createElement(vdom, nodeFilter = () => true) { let node; if (vdom.type === 'VirtualText') { node = document.createTextNode(vdom.text); } else { node = typeof vdom.namespace === 'undefined' ? document.createElement(vdom.tagName) : document.createElementNS(vdom.namespace, vdom.tagName); for (let name in vdom.attributes) { node.setAttribute(name, vdom.attributes[name]); } vdom.children.forEach((cnode) => { const childNode = createElement(cnode, nodeFilter); if (childNode && nodeFilter(childNode) ) { node.appendChild(childNode); } }); } if (vdom.__flow) { node.__flow = vdom.__flow; } return node;}

Here, we use the API: MutationObserver. What's even more gratifying is that this API is compatible with all browsers, so we can use it boldly.

Using MutationObserver:

 const options = { childList: true, // Whether to observe changes in child nodes subtree: true, // Whether to observe changes in all descendant nodes attributes: true, // Whether to observe changes in attributes attributeOldValue: true, // Whether to observe changes in attributes The changed old value characterData: true, // Whether the node content or node text changes characterDataOldValue: true, // Whether the node content or node text changes the old value // attributeFilter: ['class', 'src'] Properties not in this array will be ignored when changing};const observer = new MutationObserver((mutationList) => { // mutationList: array of mutation});observer.observe(document.documentElement, options);

It is very simple to use. You only need to specify a root node and some options that need to be monitored. Then when the DOM changes, there will be a mutationList in the callback function, which is a list of DOM changes. The structure of the mutation is roughly:

 { type: 'childList', // or characterData, attributes target: <DOM>, // other params}

We use an array to store mutations. The specific callback is:

 const onMutationChange = (mutationsList) => { const getFlowId = (node) => { if (node) { // The newly inserted DOM has no mark, so it needs to be compatible here if (!node.__flow) node.__flow = { id: uuid() }; return node.__flow.id; } }; mutationsList.forEach((mutation) => { const { target, type, attributeName } = mutation; const record = { type, target: getFlowId(target), }; switch (type) { case 'characterData': record.value = target.nodeValue; break; case 'attributes': record.attributeName = attributeName; record. attributeValue = target.getAttribute(attributeName); break; case 'childList': record.removedNodes = [...mutation.removedNodes].map(n => getFlowId(n)); record.addedNodes = [...mutation.addedNodes].map((n) => { const snapshot = this.takeSnapshot(n); return { ...snapshot, nextSibling: getFlowId(n. nextSibling), previousSibling: getFlowId(n.previousSibling) }; }); break; } this.records.push(record); });}function takeSnapshot(node, options = {}) { this.markNodes(node); const snapshot = { vdom: createVirtualDom(node), }; if (options.doctype === true) { snapshot.doctype = document.doctype.name; snapshot.clientWidth = document.body.clientWidth; snapshot.clientHeight = document.body.clientHeight; } return snapshot;}

You only need to note here that when you process a new DOM, you need an incremental snapshot. Here you still use Virtual DOM to record. When you play it later, you still generate the DOM and insert it into the parent element, so here You need to refer to the DOM, which is the sibling node.

The above MutationObserver cannot monitor the value changes of input and other elements, so we need to perform special processing on the values ​​of form elements.

MDN documentation: https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHandlers/oninput

Event objects: select, input, textarea

 window.addEventListener('input', this.onFormInput, true);onFormInput = (event) => { const target = event.target; if ( target && target.__flow && ['select', 'textarea', 'input' ].includes(target.tagName.toLowerCase()) ) { this.records.push({ type: 'input', target: target.__flow.id, value: target.value, }); }}

Use capture to capture events on the window. This is also done later. The reason for this is that we may and often prevent bubbling during the bubbling stage to implement some functions, so using capture can reduce event loss. In addition, like scroll events It will not bubble and must be captured.

MDN documentation: https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHandlers/oninput

The input event cannot satisfy the monitoring of type checkbox and radio, so it is necessary to use the onchange event for monitoring.

 window.addEventListener('change', this.onFormChange, true);onFormChange = (event) => { const target = event.target; if (target && target.__flow) { if ( target.tagName.toLowerCase() == = 'input' && ['checkbox', 'radio'].includes(target.getAttribute('type')) ) { this.records.push({ type: 'checked', target: target.__flow.id, checked: target.checked, }); } }}

MDN documentation: https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHandlers/onfocus

 window.addEventListener('focus', this.onFormFocus, true);onFormFocus = (event) => { const target = event.target; if (target && target.__flow) { this.records.push({ type: 'focus ', target: target.__flow.id, }); }}

MDN documentation: https://developer.mozilla.org/en-US/docs/Web/API/GlobalEventHandlers/onblur

 window.addEventListener('blur', this.onFormBlur, true); onFormBlur = (event) => { const target = event.target; if (target && target.__flow) { this.records.push({ type: 'blur ', target: target.__flow.id, }); }}

This refers to audio and video. Similar to the form elements above, you can monitor onplay, onpause events, timeupdate, volumechange and other events, and then store them in records.

No event is thrown when the canvas content changes, so we can:

Collect canvas elements and regularly update real-time content. Hack some drawing APIs to throw events.

The research on canvas monitoring is not very in-depth and further in-depth research is needed.

The idea is relatively simple, just get some information from the backend:

• Full snapshot Virtual DOM
• Operation chain records
• screen resolution
• doctype

Using this information, you can first generate the page DOM, which includes filtering script tags, then create an iframe and append it to a container, which uses a map to store the DOM

 function play(options = {}) { const { container, records = [], snapshot ={} } = options; const { vdom, doctype, clientHeight, clientWidth } = snapshot; this.nodeCache = {}; this.records = records; this.container = container; this.snapshot = snapshot; this.iframe = document.createElement('iframe'); const documentElement = createElement(vdom, (node) => { // Cache DOM const flowId = node.__flow && node.__flow.id; if (flowId) { this.nodeCache[flowId] = node; } // Filter script return !(node.nodeType == = Node.ELEMENT_NODE && node.tagName.toLowerCase() === 'script'); }); this.iframe.style.width = `${clientWidth}px`; this.iframe.style.height = `${clientHeight}px`; container.appendChild(iframe); const doc = iframe.contentDocument; this.iframeDocument = doc; doc.open(); doc.write(`<!doctype ${doctype}><html><head></head><body></body></html>`); doc.close(); doc.replaceChild(documentElement, doc.documentElement); this.execRecords();} 

 function execRecords(preDuration = 0) { const record = this.records.shift(); let node; if (record) { setTimeout(() => { switch (record.type) { // 'childList', 'characterData' , // Handling of 'attributes', 'input', 'checked', // 'focus', 'blur', 'play''pause' and other events} this.execRecords(record.duration); }, record.duration - preDuration) }}

The above duration is omitted in the above article. You can optimize the playback smoothness according to your own optimization, and see whether multiple records are presented as one frame or as they are.

The above is the entire content of this article. I hope it will be helpful to everyone’s study. I also hope everyone will support VeVb Wulin Network.