Know how to set up .Htaccess and robots.txt
.htaccess file – This is a file that starts with a dot. This file is used to manage server level configurations that comes to handling redirections and permissions on high level.
Wikipedia definition for htaccess file:
An .htaccess file is a directory-level configuration file supported by several web servers, used for configuration of website-access issues, such as URL redirection, URL shortening, access control, and more. The ‘dot’ before the file name makes it a hidden file in Unix-based environments.
When you change the permalink structure of your blog from settings page, a new htaccess file is automatically created so you don’t need to worry.
The default post structure like “domainname.com/?p=123” do not need an htaccess file, but if you change it like “domainname.com/sample-post/” then an htaccess file is created automatically by WordPress in the folder where it is installed.
As a basic WordPress user, you would only need to use .htaccess during the WordPress installation which happens automatically.
Robots.txt file – Robots.txt file is used by search engines to honor your request of scanning/indexing files within your site and whether those files/folders should be considered public or private. This is not a mandatory file, but you need to have this simple .txt file if you are running multiple sub directories inside your WordPress blog directory or folder AND you need to maintain different permissions or visibility for those files when it comes to search engines.
If you have multiple non-WordPress files or folders and you need to have them private, then you may need to use the robots.txt
You can use the allow/Disallow configurations accordingly as explained here